PERG-Rx

The Markov chain embedding technique is commonly used to study the distribution of statistics associated with regular patterns (i.e. set of strings described by a regular expression) in random strings. In this extended abstract, we formalize the concept Markov chain embedding for random strings produced by a possibly non-stationary Markov source. A notion of memory conveyed by the states of a deterministic finite automaton is introduced. This notion is used to characterize the smallest state-space size Markov chain required to specify the distribution of the count statistic of a given regular pattern. The research finds applications in problems associated with regular patterns in random strings that demand exponentially large state spaces.

This cookbook provides more than 100 recipes to help you crunch data and manipulate text with regular expressions. Every programmer can find uses for regular expressions, but their power doesn't come worry-free. Even seasoned users often suffer from poor performance, false positives, false negatives, or perplexing bugs. Regular Expressions Cookbook offers step-by-step instructions for some of the most common tasks involving this tool, with recipes for C#, Java, JavaScript, Perl, PHP, Python, Ruby, and VB.NET. With this book, you will: * Understand the basics of regular expressions through a concise tutorial * Use regular expressions effectively in several programming and scripting languages * Learn how to validate and format input * Manage words, lines, special characters, and numerical values * Find solutions for using regular expressions in URLs, paths, markup, and data exchange * Learn the nuances of more advanced regex features * Understand how regular expressions' APIs, syntax, and behavior differ from language to language * Write better regular expressions for custom needs Whether you're a novice or an experienced user, Regular Expressions Cookbook will help deepen your knowledge of this unique and irreplaceable tool. You'll learn powerful new tricks, avoid language-specific gotchas, and save valuable time with this huge library of proven solutions to difficult, real-world problems.

We propose quantitative regular expressions (QREs) as a high-level programming abstraction for specifying complex numerical queries over data streams in a modular way. Our language allows the arbitrary nesting of orthogonal sets of combinators: (a) generalized versions of choice, concatenation, and Kleene-iteration from regular expressions, (b) streaming (serial) composition, and (c) numerical operators such as min, max, sum, difference, and averaging. Instead of requiring the programmer to figure out the low-level details of what state needs to be maintained and how to update it while processing each data item, the regular constructs facilitate a global view of the entire data stream splitting it into different cases and multiple chunks. The key technical challenge in defining our language is the design of typing rules that can be enforced efficiently and which strike a balance between expressiveness and theoretical guarantees for well-typed programs. We describe how to compile each QRE into an efficient streaming algorithm. The time and space complexity is dependent on the complexity of the data structure for representing terms over the basic numerical operators. In particular, we show that when the set of numerical operations is sum, difference, minimum, maximum, and average, the compiled algorithm uses constant space and processes each symbol in the data stream in constant time outputting the cost of the stream processed so far. Finally, we prove that the expressiveness of QREs coincides with the streaming composition of regular functions, that is, MSO-definable string-to-term transformations, leading to a potentially robust foundation for understanding their expressiveness and the complexity of analysis problems.

Background SAS has a set of RX (“regular expression”) pattern matching and string manipulation functions. They provide a great deal more flexibility and power than string functions such as INDEX(), SCAN(), SUBSTR(), etc. in managing character strings. These functions enable one to locate, extract, and change patterns of character strings and are especially useful in situations of varying data patterns, formats, and placement within your source data.

Generation of Pattern-Matching Algorithms by Extended Regular Expressions

The main objective of online anomaly detection is to identify abnormal/unusual behavior such as network intrusions, malware infections, over utilized system resources due to design defects etc from real time data stream. Terrabytes of performance data generated in cloud data centers is a well accepted example of such data stream in real time. In this paper, we propose an online anomaly detection framework using non-parametric statistical technique in cloud data center. In order to determine the accuracy of the proposed work, we experiments it to data collected from RUBis cloud testbed and Yahoo Cloud Serving Benchmark (YCSB). Our experimental results shows the greater accuracy in terms of True Positive Rate (TPR), False Positive Rate (FPR), True Negative Rate (TNR) and False Negative Rate (FNR).

Many applications require searching for multiple patterns in large data streams for which there is no preprocessed index to rely on for efficient lookups. An multiple instruction stream-single data stream (MISD) VLSI architecture that is based on a recursive divide and conquer approach to pattern matching is proposed. This architecture allows searching for multiple patterns simultaneously. The patterns can be constructed much like regular expressions, and add features such as requiring subpatterns to match in a specific order with some fuzzy distance between them, and the ability to allow errors according to prescribed thresholds, or ranges of such. The current implementation permits up to 127 simultaneous patterns at a clock frequency of 100 MHz, and does 1.024/spl times/10/sup 11/ character comparisons per second.

When match against Chinese keyword for network content audit, one of the biggest problems is that there is interference of ¿noise characters¿, it makes the traditional way using explicit string pattern to match infeasible. Regular expression matching can solve the problem perfectly, but the DFA-base approaches for regular expression matching will also encounter the problem of excessive memory usage. In this paper, we try to solve the problem encountered when applying regular expression to Chinese network content audit. We propose a regular expression rewriting techniques and grouping principle that can solve excessive memory usage problem in DFA-based approach. Our solution can make it possible to apply regular expression to Chinese network content audit.

Anomaly detection over multi-dimensional data stream has attracted considerable attention recently in various fields, such as network, finance and aerospace. In many cases, anomalies are composed of a sequence of multi-dimensional data, and it's necessary to detect this type of anomalies accurately and efficiently over data stream. Existing online methods of anomaly detection merely focus on the single-dimensional sequence. What's more, current studies about multi-dimensional sequence are mainly concentrated on static database. However, the anomaly detection for multi-dimensional sequence over data stream is much more difficult, due to the complexity of multidimensional sequence processing, the dynamic nature of data stream and the unbalance between normal and abnormal data. Facing these challenges, we propose an anomaly detection method for multi-dimensional sequence over data stream based on cost sensitive support vector machine (C-SVM) called ADMS. First, to improve the accuracy and efficiency, the ADMS transforms multi-dimensional sequences into feature vectors in a lossless way and prunes worthless features of these vectors. And then, the ADMS can detect abnormal sequences over dynamically imbalanced data stream by lively testing these vectors based on C-SVM. Experiments show that the false negative rate (FNR) of the ADMS is lower than 5%, the false positive rate (FPR) is lower than 7%, and the throughput is improved 42% by pruning worthless features. In addition, the AMDS performs well when there are concept drifts over the data stream.

In temperate regions rotavirus diarrhea is a disease of the cooler months of the year, but little is known about its patterns in the summer. We report on the first year of national surveillance of rotavirus, during which we actively investigated patterns of summer activity. We obtained data on rotavirus testing from 85 laboratories in 48 states, conducted a survey of their testing practices and retested for confirmation positive specimens from laboratories reporting high rates of positivity during the summer. During 1989 participating laboratories reported 4011 specimens tested for rotavirus during July and August, of which 436 (11%) were said to be positive. Most laboratories reported low rates of positivity during these months (median percent positive, 3), but five had very high rates of summer positivity (> 30%). These five laboratories were geographically separated, and neighboring laboratories showed little rotavirus activity. Positive specimens submitted by four of these centers with high rates of summer rotavirus could not be confirmed. A survey of laboratory methods found one commercial assay (TestPack) and two laboratory practices (failure to use controls and involvement of more than six technicians in the testing process) to be associated with high rates of summer positivity. Moderate rates of positivity (11 to 30%) were fond frequently in the southwest during July and August; reference testing of specimens from these laboratories confirmed positivity.(ABSTRACT TRUNCATED AT 250 WORDS)

Introduction Most electronically stored endoscopic reports consist of semi-structured free text. containing a significant amount of auditable information. However, the unstandardised text requires cleaning prior to audit so that the data is standardised across patients. One major obstruction to the automated auditing of free text is phrases that mention an absence of a pathology such as ‘No hiatus hernia is seen.’ or ‘There is no malignancy.’ Methods The current study aims to determine the complexity of semi-structured free text endoscopy reports and determine the simplest function in the globally most popular data analytics language ‘R’, to remove negative phrases from endoscopic text and to validate the performance of this algorithm against manual extraction. Five hundred endoscopic reports were randomly selected for any procedure between Jan 1 2007 to Jan 1 2017. The low average text complexity (measured using the package ‘readability’ in R based on sentence structure rather than terminology) was equivalent to age 14–15 years (Readability scores: Flesch KinCaid=9.8, Gunning_Fog Index=13.3, Coleman_Liau=10.0) meaning that the text could be analysed using less processor intensive pattern recognition defined as ‘regular expressions ‘rather than using machine learning. Negative phrases were extracted from the first 100 reports as a training set, using the ‘lexicon’ package in R by cross referencing each sentence in an endoscopy report with a list of known negators to create generic regular expressions that could detect all negative phrases and incorporated into a script in R. A endoscopist blinded to the automated results was required to remove all negative phrases whether in mixed or non-mixed sentences for the same 400 endoscopy reports. Automated and manually extracted phrases were then statistically compared to generate a sensitivity, specificity and accuracy measurement of automated extraction. The difference in audit outcome was also estimated when a comparison was made between simple keyword searches to populate the audit versus the use of negative removal as implemented using our technique. Results The training set resulted in 12 regular expressions, compiled into a single runnable function, thought capable of detecting most negative phrases. This resulted in a sensitivity for the detection of negative phrases of 97.2%, specificity 74.61%, positive prediction value of 65.4% and a negative prediction value of 98.01%. When the regular expression function was run against a simple keyword search for certain phrases, certain terms demonstrated a significant difference in their detection when keyword searches were used before and after the negator removal function was used.The term CLO for example was detected 12.25% without negator removal and 7.25% with the function. Conclusions Given the simple sentence structure of the endoscopy reports across multiple endoscopists, a simple processor-nonintensive approach to the removal of negative or normal findings is feasible. Although there is room to improve the specificity further, it is preferable to have false positives in the output result which may require minimal work to remove in the final data cleaning stage. The removal of negators is likely to have more or less of an impact based on the phrase being searched for.

Data streams, or data sets which continuously and rapidly grow over time, are a prominent form of clinical data generated during the monitoring and treatment of patients in the health care industry. We propose the name Health Data Stream Analytics (HDSA) to the application of stream data processing to clinical data. Our work in this area is demonstrating the useful role Health Data Stream Analytics can play in clinical decision support, patient safety improvement and early detection of adverse patient outcomes. Two major challenges in applying stream data processing to heath care are tailoring query support for the clinical context and dealing with the clinical requirement of online query processing. In this paper, we propose the Anaesthetic Data Analyser (ADA) as a Health Data Stream Analytics System for the anaesthetics specialty and describe how it addresses these challenges. ADA differentiates from current approaches by looking at trends in the data stream rather than a single data value against a preset threshold. The trend analysis supported by ADA is a novel application in this area, and enables support for adverse symptoms monitoring in physiological stream data, alerting clinicians when a pre-defined adverse data pattern is detected in the physiological signals. This paper also describes an online query processing algorithm and the results of experiments on “real world” physiological steam data which indicate the algorithms has sub-second response times for trend queries.

In this paper, we propose a systolic pattern-independent hardware regular expression matching (REM) engine which handles nested Kleene operators used in virus patterns. Pattern-independent systolic REM engines are suitable to network intrusion detection systems for quick updating of virus pattern. In the proposed engine, we introduce a compact pattern-independent NFA circuit, which can handle any small regular expression patterns, into a systolic REM engine to handle nested Kleene operators. Experimental results show that the extended engine implemented on an FPGA handles nested Kleene operators with efficient circuit size and high performance (2.17Gbps).

Objective To evaluate the performance of three antibody kits for novel coronavirus (SARS-CoV-2) and to investigate the feasibility and advantages of them in clinical application. Methods A total of 104 patients who were admitted to Guangzhou Eighth People's Hospital with COVID-19 from January to February 2020 were selected as research group. Fifty-one healthy subjects were selected during the same period as negative control group. Serum antibodies (IgM/IgG) against SARS-CoV-2 were detected using two kinds of colloidal gold kits (A and B kits) and one chemiluminescence kit (C kit). The positive rates of SARS-CoV-2 nucleic acid in different samples from patients with COVID-19 were retrospectively analyzed. Results The clinical sensitivity of A kit to detect SARS-CoV-2-specific IgM and IgG was 77.88% (81/104) and 65.38% (68/104), respectively, and the clinical specificity was 70.59% (36/51) and 100.00% (51/51). However, the false positive rate in IgM detection was as high as 29.41% (15/51). The sensitivity of B kit to test total antibodies to SARS-CoV-2 was 63.46% (66/104), and the clinical specificity was 94.12% (48/51). The clinical sensitivity of C kit to detect SARS-CoV-2-specific IgM and IgG were respectively 31.73% (33/104) and 64.42% (67/104), and the clinical specificity were both 98.04% (50/51). There was a moderate correlation between the detection results of two colloidal gold kits and the chemiluminescence kit with the Kappa values of 0.462 and 0.587 (Z=6.157, P<0.01; Z=7.345, P<0.01). C kit had the highest positive detection rate for IgG, and would be more reliable to be used for IgG detection in COVID-19 patients 14 d after onset. The total positive detection rate of nucleic acid in all types of samples was 63.46% (66/104). The highest positive detection rate was in throat swabs or sputum samples, followed by those in blood samples and anal swabs. No viral nucleic acid was detected in urine samples for the time being. Conclusions SARS-CoV-2-specific antibodies could be detected in the early or late stage of COVID-19. The method of antibody detection has the advantages of shorter detection time, simple operation and high biological safety, indicating that it could be used as a supplementary or auxiliary detection for the diagnosis of suspected COVID-19 cases with negative nucleic acid test results. The chemiluminescence kit has good sensitivity and specificity, and is well recommended for clinical laboratories. Key words: SARS-CoV-2; Chemiluminescence immunoassay; Colloidal gold immunoaasay; Nucleic acid detection; Performance evaluation; Clinical application

Bloom filter and its variants are elegant space-efficient probabilistic data structures for approximate set membership queries. It has been recently shown that the space cost of Bloom filters can be significantly reduced via a combination with pre-trained machine learning models, named Learned Bloom filters (LBF). LBF eases the space requirement of a Bloom filter by undertaking part of the queries using a classifier. However, current LBF structures generally target a static member set. Their performances would inevitably decay when there is a member update on the set, while this update requirement is not uncommon for real-world data streaming applications such as duplicate item detection, malicious URL checking, and web caching. To adapt LBF to data streams, we propose the Stable Learned Bloom Filters (SLBF) which addresses the performance decay issue on intensive insertion workloads by combining classifier with updatable backup filters. Specifically, we propose two SLBF structures, Single SLBF (s-SLBF) and Grouping SLBF (g-SLBF). The theoretical analysis on these two structures shows that the expected false positive rate (FPR) of SLBF is asymptotically a constant over the insertion of new members. Extensive experiments on real-world datasets show that SLBF introduces a similar level of false negative rate (FNR) but yields a better FPR/storage trade-off compared with the state-of-the-art (non-learned) Bloom filters optimized on data streams.