Data Preprocessing

Harassing and fraud calls have spread like viruses in people's lives, many researchers have proposed some solutions to abnormal phone detection. However, most of these methods are passive detection, cannot give accurate prediction in time. In this work, we worked with operators to obtain a volume of real telecom user data and extract a series of comprehensive features. We propose an integration method of classifiers for abnormal phone detection by applying the machine learning algorithm on the data with unbalance and ‘dirty data’. Especially, we use bootstrap sampling method and voting strategy to reduce the false prediction of classier due to noise data. The experimental result shows the effectiveness of our method in contrast with traditional classification algorithm.

Robust PCA, the problem of PCA in the presence of outliers has been extensively investigated in the last few years. Here we focus on Robust PCA in the outlier model where each column of the data matrix is either an inlier or an outlier. Most of the existing methods for this model assumes either the knowledge of the dimension of the lower dimensional subspace or the fraction of outliers in the system. However in many applications knowledge of these parameters is not available. Motivated by this we propose a parameter free outlier identification method for robust PCA which a) does not require the knowledge of outlier fraction, b) does not require the knowledge of the dimension of the underlying subspace, c) is computationally simple and fast d) can handle structured and unstructured outliers. Further, analytical guarantees are derived for outlier identification and the performance of the algorithm is compared with the existing state of the art methods in both real and synthetic data for various outlier structures.

It is expected the predictive performance of genomic prediction methods may be adversely affected in the presence of outliers. In agriculture science an outlier may arise due to wrong data imputation, outlying response, and in a series of trials over the time or location. Although several statistical procedures are already there in literature for identification of outlier but identification of true outlier is still a challenge especially in case of high dimensional genomic data. Here we have proposed an efficient approach for detecting outlier in high dimensional genomic data, our approach is p-value based combination methods to produce single p-value for detecting the outliers. Robustness of our approach has been tested using simulated data through the evaluation measures like precision, recall etc. It has been observed that significant improvement in the performance of genomic prediction has been obtained by detecting the outliers and handling them accordingly through our proposed approach using real data.

In recent years, data quality issues have attracted wide attention. Data quality is mainly caused by dirty data. Currently, many methods for dirty data management have been proposed, and one of them is entity-based relational database in which one tuple represents an entity. The traditional query optimizations having the ability to estimate the cost of execution of a query plan have not been suitable for the new entity-based model. Then new query optimizations need to be developed. In this paper, we propose new query selectivity estimation based on histogram, and focus on solving the overestimation which traditional methods lead to. We prove our approaches are unbiased. The experimental results on both real and synthetic data sets show that our approaches can give good estimates with low error.Keywordsquery estimationhistogramdirty datadata quality

Although the kriged Kalman filter (KKF) has well-documented merits for prediction of spatial-temporal processes, its performance degrades in the presence of outliers due to anomalous events, or measurement equipment failures. This paper proposes a robust KKF model that explicitly accounts for presence of measurement outliers. Exploiting outlier sparsity, a novel l1-regularized estimator that jointly predicts the spatial-temporal process at unmonitored locations, while identifying measurement outliers is put forth. Numerical tests are conducted on a synthetic Internet protocol (IP) network, and real transformer load data. Test results corroborate the effectiveness of the novel estimator in joint spatial prediction and outlier identification.

As for the ultra high performance to determine unique earth gravity field model and its geoid, systematic errors and existing outliers need to be removed from the satellite gravimetry observation before scientific product process. In this work, we introduced an improved pre-processing algorithm for satellite gravimetry data. Firstly, scale-factors of observations are calibrated based on certain regional terrestrial-gravity data. Then on the basis of wavelet theory, an outlier-detection algorithm for satellite gravity gradiometry by applying a wavelet de-noising method to some simulation data with white noise and outliers is proposed. The computation result shows that this novel algorithm has a 97 % success rate in outlier identification and that it can be efficiently used for pre-processing real Satellite Gravity Gradiometry data.

Theory and observation show that glacier-flow regimes characterized by high basal slip enhance the projection of topographic detail to the surface, motivating this investigation into the efficacy of using glacier surges to improve bed estimation. Here we adapt a Bayesian inversion scheme and apply it to real and synthetic data as a proof of concept. Synthetic tests show a reduction in mean RMSE between true and inferred beds by more than half, and an increase in the mean correlation coefficient of ~0.5, when data from slip- versus deformation-dominated regimes are used. Multi-epoch inversions, which partition slip- and deformation-dominated regimes, are shown to outperform inversions that average over these flow regimes thereby squandering information. Tests with real data from a surging glacier in Yukon, Canada, corroborate these results, while highlighting the challenges of limited or inconsistent data. With the growing torrent of satellite-based observations, fast-flow events such as glacier surges offer potential to improve bed estimation for some of the world's most dynamic glaciers.

PreviousNext No AccessSEG Technical Program Expanded Abstracts 2001Analysis of the slip sweep techniqueAuthors: Julien MeunierPascal NicodemeSalvador RodriguezJulien MeunierCompagnie Générale de Géophysique, Pascal NicodemeTotalFinaElf, and Salvador RodriguezInstitut Français du Pétrolehttps://doi.org/10.1190/1.1816810 SectionsAboutPDF/ePub ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinked InReddit Permalink: https://doi.org/10.1190/1.1816810FiguresReferencesRelatedDetailsCited byReferences21 March 2012 SEG Technical Program Expanded Abstracts 2001ISSN (print):1052-3812 ISSN (online):1949-4645Copyright: 2001 Pages: 2135 publication data© 2001 Copyright © 2001 Society of Exploration GeophysicistsPublisher:Society of Exploration Geophysicists HistoryPublished Online: 03 Jan 2005 CITATION INFORMATION Julien Meunier, Pascal Nicodeme, and Salvador Rodriguez, (2001), "Analysis of the slip sweep technique," SEG Technical Program Expanded Abstracts : 9-12. https://doi.org/10.1190/1.1816810 Plain-Language Summary PDF DownloadLoading ...

PurposeIn recent years, the frequency and severity of cybersecurity incidents have prompted customers to seek out specialized insurance products. However, this has also presented insurers with operational challenges and increased costs. The assessment of risks for health systems and cyber–physical systems (CPS) necessitates a heightened degree of attention. The significant values of potential damages and claims request a solid insurance system, part of cyber-resilience. This research paper focuses on the emerging cyber insurance market that is currently in the process of standardizing and improving its risk analysis concerning the potential insured entity.Design/methodology/approachThe authors' approach involves a quantitative analysis utilizing a Likert-style questionnaire designed to survey cyber insurance professionals. The authors' aim is to identify the current methods used in gathering information from potential clients, as well as the manner in which this information is analyzed by the insurers. Additionally, the authors gather insights on potential improvements that could be made to this process.FindingsThe study the authors elaborated it has a particularly important cyber and risk components for insurance area, because it addresses a “niche” area not yet proper addressed in specialized literature – cyber insurance. Cyber risk management approaches are not uniform at the international level, nor at the insurer level. Also, not all insurers can perform solid assessments, especially since their companies should first prove that they are fully compliant with international cyber security standards.Research limitations/implicationsThis research has concentrated on analyzing the current practices in terms of gathering information about the insured entity before issuing the cyber insurance policy, level of details concerning the cyber security posture of the insured entity and way such information should be analyzed in a standardized and useful manner. The novelty of this research resides in the analysis performed as detailed above and the proposals in terms of information gathered, depth of analysis and standardization of approach made. Future work on the topic can focus on the standardization process for analyzing cyber risk for insurance clients, to improve the proposal based also on historical elements and trends in the market. Thus, future research can further refine the standardization process to analyze in more depth the way this can be implemented and included in relevant legislation at the EU level.Practical implicationsProposed improvements include proposals in terms of the level of detail and the usefulness of an independent centralized approach for information gathering and analysis, especially given the re-insurance and brokerage activities. The authors also propose a common practical procedural approach in risk management, with the involvement of insurance companies and certification institutions of cyber security auditors.Originality/valueThe study investigates the information gathered by insurers from potential clients of cyber insurance and the way this is analyzed and updated for issuance of the insurance policy.

داشتن درک درستی از چگونگی تغییرات زمانی و مکانی متغیر های هواشناسی در مطالعات محیطی از اهمیت بسزایی برخوردار است. اخیرا از مدل‌های منطقه ای پیش بینی عددی هواشناسی در تحقیقات بسیاری استفاده می شود. با این وجود برای بررسی چگونگی تغییرات زمانی و مکانی نتایج حاصل از مدل ها تحقیقات زیادی صورت نگرفته است. بر این اساس و با توجه به اهمیت این موضوع یک رویکرد زمانی - مکانی در این مقاله بررسی می شود. در این رویکرد نتایج پیش‌بینی مدل میان مقیاس 4MM5 با مجموعه داده های مشاهده ای، برای میانگین ماهانه و سالانه دما و بارش در دوره سال‌های 1990-2010 در منطقه شمال شرق ایران مورد تحلیل و بررسی قرار گرفت. در این تحقیق ارزیابی کمی شباهت مکانی و زمانی بین مجموعه داده‌ها و مدل آب و هوایی، و نیز شناسایی مناطق جغرافیایی و فصولی که در مدل سازی‌ها مشکل‌ساز می‌باشند انجام شده است. این کار با استفاده از نقشه‌های تهیه شده بر مبنای الگوریتم تشابه5 جبری، که یک روش موثر برای تحلیل مجموعه داده‌های مکانی مختلف، نقشه‌ها و مدل‌ها می‌باشد، انجام شد. مشخص شد که شباهت داده‌های واقعی با نتایج مدل برای متغیر دما بیش‌تر از بارندگی است. نتایج حاصل از مدل برای بارش و دما در فصل‌های گرم سال در مقایسه با داده‌های واقعی از لحاظ پراکندگی مقادیر تشابه، تناقض داشت. هم‌چنین در فصل‌های سرد سال پیش‌بینی دقیق‌تری به وسیله مدل ارائه شده است. علاوه بر این چند منطقه که در آنجا خطای پیش‌بینی بیش‌تر بود به وسیله الگوریتم نقشه‌پردازی تشابه شناسایی شدند. این تحلیل نشان داد که در محدوده‌های کوچکی از جنوب و ناحیه مرکزی منطقه مورد مطالعه بین مدل و داده‌های واقعی تشابه کم‌تری وجود دارد.

In some basins, large scale development of unconventional stacked-target plays requires early election of well targeting and spacing. Changes to the initial well construction framework can take years to implement due to lead times for land, permitting, and corporate planning. Over time, as operators wish to fine tune their development plans, completion design flexibility represents a powerful force for optimization. Hydraulic fracturing treatment plans may be adjusted and customized close to the time of investment. With a practical approach that takes advantage of physics-based modeling and data analysis, we demonstrate how to create a high-confidence, integrated well spacing and completion design strategy for both frontier and mature field development. The Dynamic Stimulated Reservoir Volume (DSRV) workflow forms the backbone of the physics-based approach, constraining simulations against treatment, flow-back, production, and pressure-buildup (PBU) data. Depending on the amount of input data available and mechanisms investigated, one can invoke various levels of rigor in coupling geomechanics and fluid flow – ranging from proxies to full iterative coupling. To answer spacing and completions questions in the Denver Basin, also known as the Denver-Julesburg (DJ) Basin, we extend this modeling workflow to multi-well, multi-target, and multi-variate space. With proper calibration, we are able generate production performance predictions across the field for a range of subsurface, well spacing, and completion scenarios. Results allow us to co-optimize well spacing and completion size for this multi-layer column. Insights about the impacts of geology and reservoir conditions highlight the potential for design customization across the play. Results are further validated against actual data using an elegant multi-well surveillance technique that better illuminates design space. Several elements of subsurface characterization potentially impact the interactions among design variables. In particular, reservoir fluid property variations create important effects during injection and production. Also, both data analysis and modeling support a key relationship involving well spacing and the efficient creation of stimulated reservoir volumes. This relationship provides a lever that can be utilized to improve value based on corporate needs and commodity price. We introduce these observations to be further tested in the field and models.

Despite increasing efforts during data collection, nonresponse remains sizeable in many household surveys. Statistical adjustment is hence unavoidable. By reweighting the design, weights of the respondents are adjusted to compensate for nonresponse. However, there is no consensus on how this should be carried out in general. Theoretical comparisons are inconclusive in the literature, and the associated simulation studies involve hypothetical situations not all equally relevant to reality. In this article we evaluate the three most common reweighting approaches in practice, based on real data in Norway from the two largest household surveys in the European Statistical System. We demonstrate how cross- examination of various reweighting estimators can help inform the effectiveness of the available auxiliary variables and the choice of the weight adjustment method.

Spatial or temporal aspects of neural organization are known to be important indices of how cognition is organized. However, measurements and estimations are often noisy and many of the algorithms used are probabilistic, which in combination have been argued to limit studies exploring the neural basis of specific aspects of cognition. Focusing on static and dynamic functional connectivity estimations, we propose to leverage this variability to improve statistical efficiency in relating these estimations to behavior. To achieve this goal, we use a procedure based on permutation testing that provides a way of combining the results from many individual tests that refer to the same hypothesis. This is needed when testing a measure whose value is obtained from a noisy process, which can be repeated multiple times, referred to as replications. Focusing on functional connectivity, this noisy process can be: (a) computational, for example, when using an approximate inference algorithm for which different runs can produce different results or (b) observational, if we have the capacity to acquire data multiple times, and the different acquired data sets can be considered noisy examples of some underlying truth. In both cases, we are not interested in the individual replications but on the unobserved process generating each replication. In this note, we show how results can be combined instead of choosing just one of the estimated models. Using both simulations and real data, we show the benefits of this approach in practice.

The streaming of real-time audio/video data is very challenging because of the time-varying and unreliable wireless channels, video content characteristics, limited bandwidth, dynamic topology, heterogeneous and distributed environment and high packet loss rate because of wireless interference and channel fading. Because of such open issues present in wireless networks, it is very difficult to satisfy the requirements of audio/video streaming applications such as low delay, low packet loss, jitter control etc. In such scenarios of opportunistic networks where real time data is being streamed from server node to client node, Network Coding and its variants can be used by such nodes to meet different requirements. Network coding changes the role of such multimedia nodes from store and forward to encode data packets. Encoding process includes mathematical operations on data packets. This survey paper has considered number of speculative and practical approaches and cases where network coding or its variant applied either fully or partially on multimedia traffic with the aim to improve performance and to provide protection against packet losses. This review has mainly focused on the opportunity of performance enhancement of MPEG-4 traffic over Wireless Network using Random Linear Network Coding (RLNC) with Multi Generation Mixing (MGM). Using Multi Generation Mixing, packets of greater importance has got more protection, less loss, more reconstruction and recovery of real time data.

Due to availability of computational tools for data acquisition, it is very easy to collect many dimensions from an object. Nevertheless, data acquisition from an object in an experiment may have a low number of dimensions. The analysis of low dimensional data has break-through role. But raw and sparse nature of dataset imposes new challenges and requirements for data analysis due to their special and unique characteristics. In the process of overall characterization of low-dimensional data, the data pre-processing plays crucial role. One of the first processes is normalization and standardization process. Therefore, in this paper, I would like to propose novel standardization technique called SDFS (Standardization for Distribution Free Statistics) for nonparametric data analysis. This technique is robust for small sample size with missing values of data points, which commonly exist in real time experiments lead to sparse low-dimensional data. The comprehensive experimental evaluation shows that SDFS standardization is significantly outperforms on existing standardization methods.