Text compression as rule based pattern recognition

The subject of the article is the processes of the automated analysis of server logs of software systems using Natural Language Processing methods. The goal is to develop an approach to improve the efficiency of log analysis by applying NLP algorithms and text compression methods. The tasks to be solved are: to study the features of logs as a source of technical information; to analyze modern methods of text data processing, including statistical, vector-based, and neural network approaches; to justify the feasibility of using NLP algorithms for automating log analysis; to substantiate the use of text compression methods to reduce data volume; and to develop a concept and architecture of an automated log analysis system. The methods used include: Natural Language Processing techniques such as TF-IDF, Word2Vec, FastText, and transformer-based models; text preprocessing methods; log compression and log template extraction approaches; and information retrieval methods applied to text corpora. The following results were obtained. A concept of an automated server log analysis system based on NLP methods and text compression algorithms was developed. A generalized system architecture was proposed, including modules for log collection, storage, filtering, preprocessing, text compression, NLP analysis, incident representation, and solution search. An algorithm for system operation was developed, providing step-by-step log processing while taking into account the textual nature and large volume of data. It was established that the use of NLP methods improves the accuracy of error detection and incident classification, while text compression reduces the computational load and increases system performance. Conclusions. The proposed approach improves the efficiency of software diagnostics, reduces the time required for error detection and resolution, and decreases the workload on developers. The scientific novelty of the obtained results lies in the following: an approach to automated log analysis combining NLP methods and text compression techniques was developed; a generalized architecture of a log analysis system that considers the specifics of unstructured textual data was proposed; existing log processing methods were further developed through the integration of modern NLP models with data volume optimization stages, which improves analysis efficiency in conditions of large-scale information flows

We consider grammar based text compression with longest-first substitution, where non-overlapping occurrences of a longest repeating substring of the input text are replaced by a new non-terminal symbol. We present a new text compression algorithm by simplifying the algorithm presented in S. Inenaga et al., (2003). We give a new formulation of the correctness proof introducing the sparse lazy suffix tree data structure. We also present another type of longest-first substitution strategy that allows better compression. We show results of preliminary experiments comparing grammar sizes of the two versions of the longest-first strategy and the most frequent strategy

Text compression algorithms are normally defined in terms of a source alphabet Sigma of 8-bit ASCII codes. The authors consider choosing Sigma to be an alphabet whose symbols are the words of English or, in general, alternate maximal strings of alphanumeric characters and nonalphanumeric characters. The compression algorithm would be able to take advantage of longer-range correlations between words and thus achieve better compression. The large size of Sigma leads to some implementation problems, but these are overcome to construct word-based LZW, word-based adaptive Huffman, and word-based context modelling compression algorithms.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The article examines a text compression algorithm. The research objective involves clarifying principles of the text compression process. To achieve this research objective, the author conducts an experiment during which a corpus of the English-language articles and reviews of the subject area “Shipbuilding” has been developed. Scientific originality of the study lies in the fact that the researcher for the first time discovers principles of professional vocabulary usage in scientific articles and scientific reviews. A statistical analysis allows concluding that in scientific reviews, the text compression algorithm is based on using complicated multicomponent terms. In scientific articles, two- or three-component terms prevail.

Search and Retrieval of Compressed Text

On—line text—compression algorithms are considered, where compression is done by substituting substrings of the text according to some fixed dictionary (code book). Due to the long running time of optimal compression algorithms, several on—line heuristics have been introduced in the literature. In this paper we analyse two modified version of an old algorithm introduced by Shuegraf and Heaps [9]. We will investigate the worst case behaviour of this Longest Fragment First (LFF) algorithm for several types of special dictionaries.

An ontology enhanced development kit for full text search is discussed in this paper. The development kit consists of a few software modules. All of these modules constitute a three-layer architecture. The full text layer implements the methods for full text index and search; the ontology layer implements the methods for concept definition and ontology maintenance; the user layer provides the programming APIs and config tools for programmers. This development kit uses two widely-used development kits (i.e., Apache Lucene and Jena) as its foundation. Apache Lucene provides the basic techniques for full text index and search, while Jena provides the basic ontology techniques. This development kit will be easy to use since the underlying details of implementation are encapsulated, and the performance of full text search will be improved since the ontology techniques are used to fill the gap between the concepts and the words.

The invention of Fountain codes is a major advance in the field of error correcting codes. The goal of this work is to study and develop algorithms for source and channel coding using a family of Fountain codes known as Raptor codes. From an asymptotic point of view, the best currently known sum-product decoding algorithm for non binary alphabets has a high complexity that limits its use in practice. For binary channels, sum-product decoding algorithms have been extensively studied and are known to perform well. In the first part of this work, we develop a decoding algorithm for binary codes on non-binary channels based on a combination of sum-product and maximum-likelihood decoding. We apply this algorithm to Raptor codes on both symmetric and non-symmetric channels. Our algorithm shows the best performance in terms of complexity and error rate per symbol for blocks of finite length for symmetric channels. Then, we examine the performance of Raptor codes under sum-product decoding when the transmission is taking place on piecewise stationary memoryless channels and on channels with memory corrupted by noise. We develop algorithms for joint estimation and detection while simultaneously employing expectation maximization to estimate the noise, and sum-product algorithm to correct errors. We also develop a hard decision algorithm for Raptor codes on piecewise stationary memoryless channels. Finally, we generalize our joint LT estimation-decoding algorithms for Markov-modulated channels. In the third part of this work, we develop compression algorithms using Raptor codes. More specifically we introduce a lossless text compression algorithm, obtaining in this way competitive results compared to the existing classical approaches. Moreover, we propose distributed source coding algorithms based on the paradigm proposed by Slepian and Wolf.

Reducing coding redundancy in LZW

In data compression or source coding algorithms, input sequences of symbols are converted to shorter sequences while the original information remains unchanged. One of the well-known data compression algorithms is Deflate which is designed based on the LZ method. Deflate method has three different modes where its second mode is applicable for real-time applications. In this mode, a certain static table of Huffman codes is employed during the coding procedure. In this paper, a new version of deflate algorithm is proposed and implemented in hardware. In the proposed method, a new basic coding table is employed. This table is modified adaptively based on the input sequence. Simulation results show that in this adaptive algorithm, the coding performance is improved. In the hardware implementation of the new method, through some parallelism concepts, we try to improve the hardware utilization and throughput.

A universal algorithm for sequential data compression is presented. Its performance is investigated with respect to a nonprobabilistic model of constrained sources. The compression ratio achieved by the proposed universal code uniformly approaches the lower bounds on the compression ratios attainable by block-to-variable codes and variable-to-block codes designed to match a completely specified source.

A review of automated feature recognition with rule-based pattern recognition

The goal of the research was to study the possibility of using the planned language Esperanto for text compression, and to compare the results of the text compression in Esperanto with the compression in natural languages, represented by Polish and English. The authors performed text compression in the created program in Python using four compression algorithms: zlib, lzma, bz2, and zl4 in four versions of the text: in Polish, English, Esperanto, and Esperanto in x notation (without characters outside ASCII encoding). After creating the compression program, and compressing the proper texts, authors conducted an analysis on the comparison of compression time and the volume of the text before and after compression. The results of the study confirmed the hypothesis, based on which the planned language, Esperanto, gives better text compression results than the natural languages represented by Polish and English. The confirmation by scientific methods that Esperanto is more optimal for text compression is the scientific added value of the paper.

A prototype chip, called RD53A, has been designed by the RD53 collaboration to face the very high hit and trigger rate requirements (up to 3 GHz/cm2 and 1 MHz, respectively) of the High Luminosity LHC experiment upgrades. In this paper, an improved algorithm for data compression, capable of sustaining the very high data volume and proposed to be implemented in the periphery of the chip, is presented: it exploits Run Length Encoding (RLE) and Variable Length Coding (VLC) to compact chip pixel hit patterns. The compression and decompression algorithms are implemented with MATLAB, and the performance is calculated taking into account the RD53A data readout implementation and its chip simulation and verification framework (called VEPIX53). In all considered cases, the results show that the RLE and VLC combination achieves a data compression ratio between 1.57 and 1.62, resulting in a bitstream size reduction between 36.2% and 38.4% with respect to the rate of the current data transmission format.

In this study, the authors present an energy-efficient data compression protocol for data collection in wireless sensor networks (WSNs). WSNs are essentially constrained by motes’ limited battery power and networks bandwidth. The authors focus on data compression algorithms and protocol development to effectively support data compression for data gathering in WSNs. Their design of compressed data-stream protocol (CDP) is generic in the sense that other lossless or lossy compression algorithms can be easily ‘plugged’ into the proposed protocol system without any changes to the rest of the CDP. This design intends to support various different WSN applications where users may prefer more specific compression algorithms, tailored to the sensing data characteristics in question, to their general algorithm. CDP is not only able to significantly reduce energy consumptions of data gathering in multi-hop WSNs, but also able to reduce sensor network traffic and thus avoid congestion accordingly. The proposed CDP is implemented on the tinyOS platform using the nesC programming language. To evaluate their work, the authors conduct simulations via TOSSIM and PowerTOSSIM-z with real-world sensor data. The results demonstrate the significance of CDP.