Compressed full-text indexes

Fast media search and browsing is today a growing need to face the challenge of managing large collections of personal media. Traditional databases, (e.g. MySQL), or text databases (e.g. Lucene), do not address an important aspect of multimedia data: the high-dimensionality of data. In this paper, we describe the implementation and evaluation of high-dimensional data indexing structures for fast search and browsing. Index structures for high-dimensional data have been researched in the literature and several proposals exist in the literature. We compare five popular index structures on a large-scale image retrieval scenario with different visual features of varying dimensionality. Both indexing and search aspects are evaluated: indexing time, search time and the trade-off between precision and performance.

Descriptive metadata, such as an article's title, authors, institutional affiliations, keywords and date of publication, collected either manually or automatically from documents contents, is often used to search and retrieve relevant documents in an archived collection. This metadata, especially for a large text corpus such as a biomedical collection, may encapsulate patterns, trends, and other valuable information, usually revealed by using specialized data analysis software to answer specific questions. A more useful, generalized approach is to repurpose this metadata to serve as a knowledgebase to answer appropriate semantic queries.At the US National Library of Medicine (NLM), we recently archived a large biomedical collection comprising annual conference proceedings containing research findings on cholera, conducted between the years 1960-2011 under the “US-Japan Cooperative Medical Science Program” (CMSP). This program was established to address health problems in Southeast Asia and other developing countries. An R&D information management system developed at NLM, called “System for the Preservation of Electronic Resources” (SPER), automatically extracted descriptive metadata from this text corpus and built a DSpace-based archive for accessing the conference articles. SPER also used this metadata to get detailed information regarding the CMSP research community, timelines of important drugs and discoveries and international collaboration, etc., using special purpose data analysis software.In this paper, we describe the occurrence and extraction of metadata from the CMSP document set, and present an alternative approach in which this metadata is used to build a knowledgebase to support semantic queries about the CMSP Program. Specifically, we show the OWL-based hierarchical ontology model created to represent the CMSP Program with its publications, participants and international collaboration over time. We discuss the technique used to convert the extracted metadata from relational database tables to OWL/RDF assertions suitable for supporting semantic queries. We show examples of queries performed against this CMSP knowledgebase, and discuss some scalability issues. Finally we describe how this approach could be customized for other large textual collections, including one from the Food and Drug Administration previously archived by the SPER system.

Pattern matching on text data has been a fundamental field of Computer Science for nearly 40 years. Databases supporting full-text indexing functionality on text data are now widely used by biologists. In the theoretical literature, the most popular internal-memory index structures are the suffix trees and the suffix arrays, and the most popular external-memory index structure is the string B-tree. However, the practical applicability of these indexes has been limited mainly because of their space consumption and I/O issues. These structures use a lot more space (almost 20 to 50 times more) than the original text data and are often disk-resident. Ferragina and Manzini (2005) and Grossi and Vitter (2005) gave the first compressed text indexes with efficient query times in the internal-memory model. Recently, Chien et al (2008) presented a compact text index in the external memory based on the concept of Geometric Burrows-Wheeler Transform. They also presented lower bounds which suggested that it may be hard to obtain a good index structure in the external memory. In this paper, we investigate this issue from a practical point of view. On the positive side we show an external-memory text indexing structure (based on R-trees and KD-trees) that saves space by about an order of magnitude as compared to the standard String B-tree. While saving space, these structures also maintain a comparable I/O efficiency to that of String B-tree. We also show various space vs I/O efficiency trade-offs for our structures.

In this paper, we present CoMRI, compressed multiresolution index, our system for fast sequence similarity search in DNA sequence databases. We employ virtual bounding rectangle (VBR) concept to build a compressed, grid style index structure. An advantage of grid format over trees is subsequence location information is given by the order of corresponding VBR in the VBR list. Taking advantage of VBRs, our index structure fits into a reasonable size of memory easily. Together with a new optimized multiresolution search algorithm, the query speed is improved significantly. Extensive performance evaluations on human chromosome sequence data show that VBRs save 80%-93% index storage size compared to MBRs (minimum bounding rectangles) and new search algorithm prunes almost all unnecessary VBRs which guarantees efficient disk I/O and CPU cost. According to the results of our experiments, the performance of CoMRI is at least 100 times faster than MRS which is another grid index structure introduced very recently.

In this paper, we developed an auto-focus image quality evaluation function from the histogram feature function (HFF) of a high-speed camera. Based on this function, we then proposed a fast auto-focus search algorithm. We further demonstrated that the auto-focus image quality evaluation function we proposed requires fewer computational resources, while providing a more effective focusing area and more moderate sensitivity, compared to the traditional evaluation function. In contrast to the traditional climbing method, our proposed fast auto-focusing search algorithm also can spot the focus point in a shorter time, effectively reducing the round trips of the focusing motor and improving the focusing accuracy.

BackgroundSimilaritysearch in chemical structure databases is an important problem with many applications in chemical genomics, drug design, and efficient chemical probe screening among others. It is widely believed that structure based methods provide an efficient way to do the query. Recently various graph kernel functions have been designed to capture the intrinsic similarity of graphs. Though successful in constructing accurate predictive and classification models, graph kernel functions can not be applied to large chemical compound database due to the high computational complexity and the difficulties in indexing similarity search for large databases. ResultsTo bridge graph kernel function and similarity search in chemical databases, we applied a novel kernel-based similarity measurement, developed in our team, to measure similarity of graph represented chemicals. In our method, we utilize a hash table to support new graph kernel function definition, efficient storage and fast search. We have applied our method, named G-hash, to large chemical databases. Our results show that the G-hash method achieves state-of-the-art performance for k-nearest neighbor (k-NN) classification. Moreover, the similarity measurement and the index structure is scalable to large chemical databases with smaller indexing size, and faster query processing time as compared to state-of-the-art indexing methods such as Daylight fingerprints, C-tree and GraphGrep.ConclusionsEfficient similarity query processing method for large chemical databases is challenging since we need to balance running time efficiency and similarity search accuracy. Our previous similarity search method, G-hash, provides a new way to perform similarity search in chemical databases. Experimental study validates the utility of G-hash in chemical databases.

The mobile agent route is essentially a multi-constraint optimization problem, Genetic Algorithms has fast random global search ability, but the feedback information of the system does not use and has the problem of low efficiency of finding exact solutions, propose a genetic hybrid ant colony algorithm for WSN mobile agent route. Use of the fast random global search capabilities of genetic algorithm to find better solutions, then the better solution replaced by the initial pheromone of the ant colony algorithm, finally use the advantages of convergence speed of ant colony algorithm to find the global optimal solution for mobile agent route. Simulation results show that the algorithm can find optimal mobile agent route in a relatively short time, relative to other routing algorithms, reducing network latency and average energy consumption, improving the speed and efficiency of data transfer.

Document summarization provides an instrument for faster understanding the collection of text documents and has a number of real life applications. Semantic similarity and clustering can be utilized efficiently for generating effective summary of large text collections. Summarizing large volume of text is a challenging and time consuming problem particularly while considering the semantic similarity computation in summarization process. Summarization of text collection involves intensive text processing and computations to generate the summary. MapReduce is proven state of art technology for handling Big Data. In this paper, a novel framework based on MapReduce technology is proposed for summarizing large text collection. The proposed technique is designed using semantic similarity based clustering and topic modeling using Latent Dirichlet Allocation (LDA) for summarizing the large text collection over MapReduce framework. The summarization task is performed in four stages and provides a modular implementation of multiple documents summarization. The presented technique is evaluated in terms of scalability and various text summarization parameters namely, compression ratio, retention ratio, ROUGE and Pyramid score are also measured. The advantages of MapReduce framework are clearly visible from the experiments and it is also demonstrated that MapReduce provides a faster implementation of summarizing large text collections and is a powerful tool in Big Text Data analysis.

In this article, we develop a methodological approach for organizational research regarding the construction of multidimensional and relational similarity measures by using the vector space model in natural language processing (NLP). Our vector space approach draws on the well-established premise in organizational research that texts provide a window into social reality and allow measuring theory-based constructs ( e.g., organizations’ self-representations). Using a vector space approach allows capturing the multidimensionality of these theory-based constructs and computing relational similarities between organizational entities ( e.g., organizations, their members, and subunits) in social spaces and with their environments, such as the organization itself, industries, or countries. Thus, our methodological approach contributes to the recent trend in organizational research to use the potential inherent in big (textual) data by using NLP. In an example, we provide guidance for organizational scholars by illustrating how they can ensure validity when applying our methodological contribution in concrete research practice.

Users with large text collections are often faced with one of two problems; either they wish to retrieve a semanticallyrelevant subset of data from the collection for further scrutiny (needle-in-a-haystack) or they wish to glean a high-level understanding of how a subset compares to the parent corpus in the context of aforementioned semantic dimensions (forestfor-the-trees). In this paper, I describe MUCK 1 , an open-source toolkit that addresses both of these problems through a distributed text processing engine with an interactive visualization interface.

This paper proposed a fast convergence and balanced adolescent identity search algorithm (FCBAISA) for numerical and engineering design problems. The main contributions are as follows. Firstly, a hierarchical optimization strategy is proposed to balance the exploration and exploitation better. Secondly, a fast search strategy is proposed to avoid the local optimization and improve the accuracy of the algorithm; that is, the current optimal solution combines with the random disturbance of Brownian motion to guide other adolescents. Thirdly, the Chebyshev functional-link network (CFLN) is improved by recursive least squares estimation (RSLE), so as to find the optimal solution more effectively. Fourthly, the terminal bounce strategy is designed to avoid the algorithm falling into local optimization in the later stage of iteration. Fifthly, FCBAISA and comparison algorithms are tested by CEC2017 and CEC2022 benchmark functions, and the practical engineering problems are solved by algorithms above. The results show that FCBAISA is superior to other algorithms in all aspects and has high precision, fast convergence speed, and excellent performance.

We investigate privacy-preserving Public-Key Infrastructures (PKIs) with additional security, privacy, and performance requirements meant to support vehicular networks. While conventional PKIs are a cornerstone for identity management in certain security applications, they do not offer the privacy properties that vehicular networks demand. Over the last few years, there have been efforts to generalize conventional PKIs to meet these new privacy and security demands. One particular example is the class of multiple certificate PKIs where each vehicle is given multiple short-lived and anonymous certificates that can function as credentials until the vehicle is due for, say, the next inspection or dealership visit. While such PKIs offer attractive security and privacy properties, achieving acceptable performance has remained an open problem thus questioning their practicality.In this paper, we formally define and investigate the efficient certificate revocation list (CRL) search problem for multiple certificate PKIs, which asks whether it is possible to efficiently test CRL membership of a given certificate in the same order of computational complexity as in conventional PKIs (that is, by a simple and fast binary search over the set of revoked PKI members). Any solution to this problem is of particular interest when it does not introduce unacceptable compromises on other important properties such as security and privacy. We start by formally defining vehicular networks and a number of security, privacy and performance requirements for them. Then we demonstrate that for PKIs based on general classes of efficient algorithms, containing binary search over totally ordered sets and approximate nearest neighbor search over metric spaces, a solution to the efficient CRL search problem can be used to violate privacy. Finally, we present and analyze new solutions that simultaneously solve the efficient cRL search problem and satisfy privacy requirements by allowing all vehicles to locally (i.e., with minimal help from the CA) and consistently (i.e., with the same results across all vehicles) manage their CRLs. Using these ideas, we can maintain security and privacy properties while reducing the CRL membership testing time to logarithmic in the number of revoked vehicles, which is essentially the same performance that one obtains with conventional PKIs where privacy is not required.

In this article, we present a method of anaphoric proper names detection in fictional texts using Word2Vec model and algorithms of community detection on graphs. This method allows grouping different namings of a single entity and can be useful as a part of preprocessing texts for further analysis such as building social networks or training neural models. The method uses large text collection, related to the same domain. The foundation of the method is training of a Word2Vec model using information on direct characters interactions. This model allows building a social graph of characters. Than, the Louvain algorithm is used to divide the graph into communities containing different names of characters related to the same denotation.

Word-based embedding approaches such as Word2Vec capture the meaning of words and relations between them, particularly well when trained with large text collections; however, they fail to do so with small datasets. Extensions such as fastText reduce the amount of data needed slightly, however, the joint task of learning meaningful morphology, syntactic and semantic representations still requires a lot of data. In this paper, we introduce a new approach to warm-start embedding models with morphological information, in order to reduce training time and enhance their performance. We use word embeddings generated using both word2vec and fastText models and enrich them with morphological information of words, derived from kernel principal component analysis (KPCA) of word similarity matrices. This can be seen as explicitly feeding the network morphological similarities and letting it learn semantic and syntactic similarities. Evaluating our models on word similarity and analogy tasks in English and German, we find that they not only achieve higher accuracies than the original skip-gram and fastText models but also require significantly less training data and time. Another benefit of our approach is that it is capable of generating a high-quality representation of infrequent words as, for example, found in very recent news articles with rapidly changing vocabularies. Lastly, we evaluate the different models on a downstream sentence classification task in which a CNN model is initialized with our embeddings and find promising results.

In Russian, Chinese and English, there is a variety of vocabulary related to taste sensations, which often has a deep figurative meaning. The subject of this study is precisely such a taste vocabulary in the mentioned languages. The object of the research is cognitive metaphors of key taste words, which are used in creating emotional images-schemes. The aim of the study is to study and compare the similarities and differences in the use of cognitive metaphors for the most important taste words in emotional images—schemes typical of Russian, Chinese and English languages. The main task of the study is to analyze the ways of metaphorical representation of taste vocabulary in these emotional images-schemes and identify the unique features of each language in the context of this topic. The research methods are comparative analysis and the corpus method, which allows you to analyze the use of language in large text collections. The novelty of this study lies in the development of a methodology for the analysis of emotional images-schemes associated with taste words. The study shows that in Russian, Chinese and English there is a high degree of similarity in cognitive schemas for the taste words "sour" and "sweet", which most often have a similar emotional image-scheme and are located at opposite ends of the image-scheme "LINE". However, there are differences in the words "bitter" and "acute": in English, both words usually have a negative meaning, in Russian "bitter" is mostly negatively colored, and "acute" can be both negative and positive. In Chinese, both the words "bitter" and "spicy" can have both positive and negative meanings. The metaphorical representation of taste words in an emotional image-scheme helps to make the metaphorical transfer of abstract taste words more concrete and visual, which makes it possible to better understand the differences and similarities in the translation of taste sensations in different languages.