Entropy-based heavy tailed distribution transformation and visual analytics for monitoring massive network traffic

In this paper we present a new scheduler, the ?-scheduler, which performs better on heavy tailed traffic than the Foreground-Background (FB) scheduler which is known to be optimal in scheduling traffic which has unknown characteristics. The ?-scheduler is able to provide a closer approximation to the Shortest Remaining Processing Time (SRPT) scheduler which is known to provide optimal scheduling in the case when the length of the packets to be scheduled is known. We are able to improve our ?-scheduler SRPT approximation by basing our expected remaining processing time on an estimate of the shape parameter, ?, from the standard heavy tailed Pareto complementary probability distribution function, P[X > t]=ct??. We review various methods of estimating ? based on linear regression methods on the standard Log-Log Complement Distribution plot first proposed by Crovella in [5] and studied in more detail in [4]. We show that even using the standard sliding window least mean square estimator our scheduler exhibits improved performance over the FB scheduler. The particular scheduling problem we investigate in detail concerns the servicing of a number of ingress flows being fed by heavy tailed distributions. The egress channel is a bottleneck link, for example a shared wireless TDMA domain. Each flow is characterized as having a distinct shape parameter which may change (slowly) over time. While the scheduling is real time, the {?i} estimation does not need to be.

Gene Networks with Transcriptional Bursting Recapitulate Rare Transient Coordinated High Expression States in Cancer.

In this paper, heteroscedastic nonlinear regression (HNLR) models under the flexible class of two–piece distributions based on the scale mixtures of normal (TP–SMN) family were examined. This novel class of nonlinear regression (NLR) models is a generalization of the well-known heteroscedastic symmetrical nonlinear regression models. The TP–SMN is a rich class of distributions that covers symmetric and asymmetric as well as heavy-tailed distributions. Using the suitable hierarchical representation of the family, the researchers first derived an EM–type algorithm for iteratively computing maximum likelihood (ML) estimates of the parameters. Then, in order to examine the performance of the proposed models and methods, some simulation studies were presented to show the robust aspect of this flexible class against outlying and also atypical data. As the last step, a natural real dataset was fitted under the proposed HNLR models.

From observed bid and ask prices of European call and put options we estimate the risk neutral density of a stock at some future time $t>0$ . We restrict attention to a class of densities with heavy tails and use a Bayesian formulation in order to study the variation in the distributions fitting the data. Heavy tails are here meant in the intuitive sense of being heavier than the tails of a normal distribution. From the fitted risk neutral density we also consider the inverse problem of finding the volatility in a diffusion model for the price process. Finally, we apply our methods to data on the S&P 500 index.

Why does a disproportionate amount of high-growth entrepreneurship concentrate in a small number of geographical areas? Our research question is motivated by the heavy tails in the frequency-size distributions of high-growth entrepreneurship—the number of early-stage financing rounds, acquisitions among firms that received early-stage financing, and initial public offerings (IPOs)—in a metropolitan or micropolitan statistical area (MSA). The heavy tails could result from either of two distinct processes, but the existing research on entrepreneurship and agglomeration cannot tell the two processes apart. In this paper, we apply technical advances from complexity science and analyze which process is the more likely generative mechanism for the observed heavy tails. Using data from Crunchbase, our analysis covers 129,949 financing rounds of 58,517 unique firms across 657 MSAs, 10,145 acquisitions, and 4,614 IPOs. We find the multiplicative process to be the more likely mechanism, compared to the rich-get-richer process, which has been posited in the literature as a driver of regional advantage. For research, our finding changes the way we theorize and study agglomeration and entrepreneurship. For government policy, our research enhances the efficacy of interventions—in the form of tax breaks, business incubators, or grants aiming to stimulate entrepreneurial activity, spur innovation and create new jobs—by revealing the generative mechanism for the emergence of a region’s advantage in high-growth entrepreneurship. For entrepreneurs, our research informs whether and how the choice of where to locate the start-ups may have implications for the likelihood of obtaining early-stage financing, becoming acquired, and going public.

Nonlinear mixed-effects models are very useful to analyze repeated measures data and are used in a variety of applications. Normal distributions for random effects and residual errors are usually assumed, but such assumptions make inferences vulnerable to the presence of outliers. In this work, we introduce an extension of a normal nonlinear mixed-effects model considering a subclass of elliptical contoured distributions for both random effects and residual errors. This elliptical subclass, the scale mixtures of normal (SMN) distributions, includes heavy-tailed multivariate distributions, such as Student-t, the contaminated normal and slash, among others, and represents an interesting alternative to outliers accommodation maintaining the elegance and simplicity of the maximum likelihood theory. We propose an exact estimation procedure to obtain the maximum likelihood estimates of the fixed-effects and variance components, using a stochastic approximation of the EM algorithm. We compare the performance of the normal and the SMN models with two real data sets.

We use response time decision models to determine whether visual working memory (VWM) is best described by a fixed-capacity ‘slot’ model or a continuously allocated ‘resource’ model. We used a double-decision paradigm that combined continuous-outcome recall and two-choice recognition to characterize performance in a VWM task. Participants viewed one to six colored items, recalled the color of a cued target using a color wheel, and then identified the target color from two options in a recognition task. We used continuous-outcome and two-choice EZ diffusion models to analyze the speed and accuracy of the two decisions. The models provided estimates of the latent cognitive parameters, drift rate and boundary separation, with the former providing measures of the memory strengths in the two decisions. We found significant correlations in both drift rate and boundary separation for the two decisions, indicating that continuous recall and two-choice recognition tasks engage similar memory and decision processes. Removing responses within the highly inaccurate ‘heavy tails’ of the recall distribution reduced, rather than increased, drift rate correlations – especially in the six item condition – implying that the heavy tails of the response distribution represent meaningful low-precision memory traces rather than guesses. Simulations showed that only a small fraction of the heavy tailed memory trace could be accounted for by swap errors. Our results are consistent with variable precision and resource accounts of VWM and inconsistent with slot-based accounts. Our findings underscore the value of joint modeling response time and accuracy data, and highlight the utility of double-decision tasks in clarifying theoretical accounts of memory processes.

The paper is part of a larger paper (see Hernandez-Campos, F. et al., http://www.cs.unc.edu/Research/dirt/proj/marron/VarHeavyTails, 2002) that studies the tails of the duration distribution of Internet data flows, and their heaviness. Data analysis motivates the concepts of moderate, far and extreme tails for understanding the richness of information available in the data. The analysis also motivates a notion of variable tail index, which leads to a generalization of existing theory for heavy tail durations leading to long range dependence. The emphasis here is on understanding heavy tails.

Why does a disproportionate amount of high-growth entrepreneurship concentrate in a small number of geographical areas? The heavy tails in the frequency-size distributions could result from any of four distinct processes, but the existing research on entrepreneurship and agglomeration cannot tell the four processes apart. Applying technical advances from complexity science, I identify which process is the most likely generative mechanism for the observed heavy tails. My data analysis, covering 129,949 early-stage financing rounds of 58,517 unique firms across 657 geographical areas, 10,145 acquisitions, and 4,614 initial public offerings, shows that the most likely process is incremental differentiation with diminishing returns. As suggested by this process, areas grow at different rates because of area-specific factors and the fastest growing areas have diminishing returns to scale.

Renyi entropy based design of heavy tailed distribution for return of financial assets

This paper identifies the characteristics of three-parameter Burr Type XII distribution and discusses its utility in survivorship applications. It addresses the problem of estimating the three-parameter Burr XII distribution and its doubly truncated version. The results are applied on a real dataset by fitting the distribution to the survival time of breast cancer patients in the Gaza Strip. These data are known to have a heavy tailed distribution since patients in this area received different protocols of treatments in different levels of hospitals locally and abroad. The findings indicated that the estimates of the parameters of the truncated distribution are more efficient than those obtained from the original distribution since the distribution is heavy tailed and involves many highly extreme observations.

Network traffic due to its heterogeneity can be modeled by heavy tail distributions. A 100 Mbps Ethernet link has been used to capture the network traffic (IP Packets) and the measurements on a switched Ethernet system are presented that reveal new insights into the statistical nature of the traffic. Various parameters of sniffed network traffic from six different data sets are parsed for its characterization. Packet inter-arrival time and packet size are statistically evaluated and analyzed. Packet inter-arrival time follows power law and can be modeled by heavy tail distribution (Pareto distribution). Hybrid mathematical model of packet size has also been presented

BackgroundSystematic reviews of complex interventions commonly find heterogeneity of effect sizes among similar interventions which cannot be explained. Commentators have suggested that complex interventions should be viewed as interventions in complex systems. We hypothesised that if this is the case, the distribution of effect sizes from complex interventions should be heavy tailed, as in other complex systems. Thus, apparent heterogeneity may be a feature of the complex systems in which such interventions operate.Methodology/Principal FindingsWe specified three levels of complexity and identified systematic reviews which reported effect sizes of healthcare interventions at two of these levels (interventions to change professional practice and personal interventions to help smoking cessation). These were compared with each other and with simulated data representing the lowest level of complexity. Effect size data were rescaled across reviews at each level using log-normal parameters and pooled. Distributions were plotted and fitted against the inverse power law (Pareto) and stretched exponential (Weibull) distributions, heavy tailed distributions which are commonly reported in the literature, using maximum likelihood fitting. The dataset included 155 studies of interventions to change practice and 98 studies of helping smoking cessation. Both distributions showed a heavy tailed distribution which fitted best to the inverse power law for practice interventions (exponent = 3.9, loglikelihood = −35.3) and to the stretched exponential for smoking cessation (loglikelihood = −75.2). Bootstrap sensitivity analysis to adjust for possible publication bias against weak results did not diminish the goodness of fit.Conclusions/SignificanceThe distribution of effect sizes from complex interventions includes heavy tails as typically seen in both theoretical and empirical complex systems. This is in keeping with the idea of complex interventions as interventions in complex systems.

The purpose of this paper is to survey many of the methods for estimating extreme right tail-area quantiles in order to determine which method or methods gives the best approximations. The problem is to find a good estimate of xp defined by 1 - F(x p) = p where p is a very small number for a random sample from an unknown distribution. An extension of this problem is to determine the number of largest order statistics that should be used to make an estimate. From extensive computer simulations trying to minimize relative error, conclusions can be drawn based on the value of p. For p = .02, the exponential tail method by Breiman, et al using a method by Pickands for determining the number of order statistics to use works best for light to heavy tailed distributions. For extremely heavy tailed distributions, a method proposed by Hosking and Wallis seems to be the most accurate at p = .02 and p = .002. The quadratic tail method by Breiman, et al appears best for light to moderately heavy tailed distributions at p = .002 and for all distributions at p = .0002.

Shetty and Umarani(2005) considered a special type of two-sample location problem which has some potential applications, particularly in comparing the performances of two packing machines. In this paper, a new class of test statistics is proposed which is based on sub-sample minima of first sample and sub-sample median of second sample. The performance of members of the new class is evaluated in terms of Pitman asymptotic relative efficiency in comparision with Shetty and Umarani(2005). It can be concluded that the performance of the new proposed class of tests is better for medium and heavy tailed distributions such as Normal, Logistic, Laplace and Cauchy. For light tailed distributions the performance of new testis better in specific cases. It is proved that the test is consistent for one-sided alternatives.