Compact Data Structures for the Metric Suffix Array

To utilize grid data containing location identifiers such as street names or parcel numbers on mobile devices for location verification, the data must store various location identifiers within each grid cell, resulting in large data volumes. Therefore, there is a need to reduce the storage capacity of the data. Since grid data has a structure similar to raster data, a recent approach proposes applying Compact Data Structures (CDS)-a method that simultaneously stores attribute values and indexing in a compressed format-for data compression. When the CDS-based compression method was applied to 10m grid data containing location identifiers such as road names and land-lot numbers, the storage capacity was reduced compared to the Network Common Data Form (NetCDF) format, which stores array data. Additionally, the memory usage was also reduced. Consequently, even when using grid data on mobile devices, the reduced data size allows for faster search and visualization.

Scalable and queryable compressed storage structure for raster data

Finding repetitive structures in genomes and proteins is important to understand their biological functions. Many data compressors for modern genomic sequences rely heavily on finding repeats in the sequences. The notion of maximal repeats captures all the repeats in the data in a space-efficient way. Prior work on maximal repeat finding used either a suffix tree or a suffix array along with other auxiliary data structures. Their space usage is 19--50 times the text size with the best engineering efforts, prohibiting their usability on massive data. Our technique uses the Burrows-Wheeler Transform and wavelet trees. For data sets consisting of natural language texts, the space usage of our method is no more than three times the text size. For genomic sequences stored using one byte per base, the space usage is less than double the sequence size. Our method is also orders of magnitude faster than the prior methods for processing massive texts, since the prior methods must use external memory. For the first time, our method enables a desktop computer with 8GB internal memory to find all the maximal repeats in the whole human genome in less than 17 hours. We have implemented our method as general-purpose open-source software for public use.

The objective is to develop a data structure that is capable of handling large data volumes and offers support for querying, analysis and validation. Based on earlier results (i.e. the full decomposition of space, the use of a TEN structure and applying Poincare simplicial homology as mathematical foundation) a simplicial complex-based TEN structure is developed. Applying simplicial homology offers full control over orientation of simplexes and enables one to derive substantial parts of the TEN structure, instead of explicitly store the entire network. The described data structure is developed as a DBMS data structure and the usage of views, function based indexes and 3D R-trees result in a compact topological 3D data structure. Theoretical aspects of this approach are described earlier [12, 17,18]. This paper describes both theory and implementation of the approach.

Forwarding packets based on networking names is essential for network protocols on different layers, where the `names' could be addresses, packet/flow IDs, and content IDs. For long there have been efforts using dynamic and compact data structures for fast and memory-efficient forwarding. In this work, we identify that the recently developed programmable network paradigm has the potential to further reduce the time/memory complexity of forwarding structures by separating the data plane and control plane. This work presents the new designs of network forwarding structures under the programmable network paradigm, applying three typical dynamic and compact data structures: Bloom filters, Cuckoo hashing, and Othello hashing. We conduct careful analyses and experiments in real networks of these forwarding methods for multiple performance metrics, including lookup throughput, memory footprint, construction time, dynamic updates, and lookup errors. The results give rich insights on designing forwarding algorithms with dynamic and compact data structures. In particular, the new designs based on Cuckoo hashing and Othello hashing show significant advantages over the extensively studied Bloom filter based methods, in all situations discussed in this paper.

Using Compressed Suffix-Arrays for a compact representation of temporal-graphs

In recent years, the rapid expansion of multimedia applications, partly due to the exponential growth of the Internet, has proliferated over the daily life of computer users (Yang & Hurson, 2006). The integration of wireless communication, pervasive computing, and ubiquitous data processing with multimedia database systems has enabled the connection and fusion of distributed multimedia data sources. In addition, the emerging applications, such as smart classroom, digital library, habitat/environment surveillance, traffic monitoring, and battlefield sensing, have provided increasing motivation for conducting research on multimedia content representation, data delivery and dissemination, data fusion and analysis, and contentbased retrieval. Consequently, research on multimedia technologies is of increasing importance in computer society. In contrast with traditional text-based systems, multimedia applications usually incorporate much more powerful descriptions of human thought—video, audio, and images (Karpouzis, Raouzaiou, Tzouveli, Iaonnou, & Kollias, 2003; Liu, Bao, Yu, & Xu, 2005; Yang & Hurson, 2005). Moreover, the large collections of data in multimedia systems make it possible to resolve more complex data operations such as imprecise query or content-based retrieval. For instance, the image database systems may accept an example picture and return the most similar images of the example (Cox, Miller, & Minka, 2000; Hsu, Chua, & Pung, 2000; Huang, Chang, & Huang, 2003). However, the conveniences of multimedia applications come with challenges to the existing data management schemes: • Efficiency: Multimedia applications generally require more resources; however, the storage space and processing power are limited in many practical systems, for example, mobile devices and wireless networks (Yang & Hurson, 2005). Due to the large data volume and complicated operations of multimedia applications, new methods are needed to facilitate efficient representation, retrieval, and processing of multimedia data while considering the technical constraints. • Semantic Gap: There is a gap between user perception of multimedia entities and physical representation/access mechanism of multimedia data. Users often browse and desire to access multimedia data at the object level (“entities” such as human beings, animals, or buildings). However, the existing multimedia retrieval systems tend to access multimedia data based on their lower-level features (“characteristics” such as color patterns and textures), with little regard to combining these features into data objects. This representation gap often leads to higher processing cost and unexpected retrieval results. The representation of multimedia data according to human’s perspective is one of the focuses in recent research activities; however, few existing systems provide automated identification or classification of objects from general multimedia collections. • Heterogeneity: The collections of multimedia data are often diverse and poorly indexed. In a distributed environment, because of the autonomy and heterogeneity of data sources, multimedia data objects are often represented in heterogeneous formats. The difference in data formats further leads to the difficulty of incorporating multimedia data objects under a unique indexing framework. • Semantic Unawareness: The present research on content-based multimedia retrieval is based on feature vectors—features are extracted from audio/video streams or image pixels, empirically or heuristically, and combined into vectors according to the application criteria. Because of the application-specific multimedia formats, the feature-based paradigm lacks scalability and accuracy.

In this paper, we propose a multimedia data management framework using GeM-Tree. GeM-Tree is a multidimensional tree-based index structure which provides a generalized framework to organize and retrieve images and videos seamlessly. In addition to supporting different multimedia data types and diverse representations, the proposed data management framework supports varied multimedia retrieval strategies like content-based image and video retrieval, mixed multimedia data retrieval where cross-similarity between images and videos is considered, region-based retrieval, etc. The framework embeds a high-level semantic relationship between multimedia data objects with a construct called Hierarchical Markov Model Mediator via a novel affinity promotion technique to improve query result relevance manifold. Extensive experiments were conducted with different multimedia data types possessing varied representations, different retrieval approaches, and different data sizes. The encouraging results demonstrate a potential solution to the important and complex issue of managing a large volume of multimedia data.

This paper proposes an efficient method for large-vocabulary continuous-speech recognition, using a compact data structure and an efficient algorithm. We introduce a very compact data structure DAWG as a lexicon to reduce the space. We also propose a algorithm to obtain the N-best hypotheses using the DAWG structure. This algorithm is composed of two phases: forward search and traceback. Forward search, which basically uses the time-synchronous Viterbi algorithm, merges candidates and stores the information about them in DAWG structures to create phoneme graphs. Traceback traces the phoneme graphs to obtain the N-best hypotheses. An evaluation of this method's performance using a speech-recognition-based telephone-directory-assistance system having a 4000-word vocabulary confirmed that our strategy improves speech recognition in terms of time and recognition rate.

Streaming triangle counting is a critical issue in graph stream mining, with applications in dense subgraph discovery, web mining, anomaly detection, and more. Recent efforts have focused on estimating triangle counts in graph streams, primarily through sampling methods. However, because of limited memory resources for handling high speed streams, traditional sampling methods suffer from reduced sampling rate and thereby performance loss. In this paper, we propose a new compact data structure called uHLL to process edge streams by considering the tradeoff between estimation accuracy and memory efficiency. Furthermore, different from conventional triangle counting algorithms, we solve the estimation of union set cardinality for edge-local triangle count under both centralized and distributed framework, so as to efficiently estimate the global triangle count by a one-pass streaming algorithm. To the best of our knowledge, this is the first implementation of a distributed framework using a compact data structure for streaming triangle counting. We provide theoretical proof of unbiasedness and derive the variance of the union set and global triangle count. We compare our scheme with 11 algorithms, showing that under the same experimental setting, uHLL and distributed uHLL are at least <inline-formula><tex-math notation="LaTeX">$ 2.3$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$ 1.7$</tex-math></inline-formula> times more accurate than the state-of-the-art, respectively.

Multimedia data poses challenges for efficient storage and retrieval due to its large size and playback timing requirements. For applications that store very large volumes of multimedia data, hierarchical storage offers a scalable and economical alternative to store data on magnetic disks. In a hierarchical storage architecture data is stored on a tape or optical disk based tertiary storage layer with the secondary storage disks serving as a cache or buffer. Due to the need for swapping media on drives, retrieving multimedia data from tertiary storage can potentially result in large delays before playback (startup latency) begins as well as during playback (jitter). In this paper we address the important problem of reducing startup latency and jitter for very large multimedia repositories. We propose that secondary storage should not be used as a cache in the traditional manner — instead, most of the secondary storage should be used to permanently store partial objects. Furthermore, replication is employed at the tertiary storage level to avoid expensive media switching. In particular, we show that by saving the initial segments of documents permanently on secondary storage, and replicating them on tertiary storage, startup latency can be significantly reduced. Since we are effectively reducing the amount of secondary storage available for buffering the data from tertiary storage, an increase in jitter may be expected. However, our results show that the technique also reduces jitter, in contrast to the expected behavior. Our technique exploits the pattern of data access. Advance knowledge of the access pattern is helpful, but not essential. Lack of this information or changes in access patterns are handled through adaptive techniques. Our study addresses both single- and multiple-user scenarios. Our results show that startup latency can be reduced by as much as 75% and jitter practically eliminated through the use of these techniques.

Multimedia data particularly digital videos that contain various modalities (visual, audio, and text) are complex and time consuming to deal with. Therefore, managing a large volume of multimedia data reveals the necessity for efficient methods for modeling, processing, storing and retrieving such data. In this study, we investigate how to efficiently manage multimedia data, especially video data. In addition, we discuss various flexible query types including the combination of content as well as concept-based queries that provide users with the ability to perform multimodal query. Furthermore, we introduce a fusion-based approach at the query level to improve query retrieval performance of the multimedia database. Our experimental tests show a significant improvement in the query retrieval performance.

The classic string indexing problem is to preprocess a string S into a compact data structure that supports efficient pattern matching queries. Typical queries include existential queries (decide if the pattern occurs in S), reporting queries (return all positions where the pattern occurs), and counting queries (return the number of occurrences of the pattern). In this paper we consider a variant of string indexing, where the goal is to compactly represent the string such that given two patterns \(P_1\) and \(P_2\) and a gap range \({[}\alpha , \beta ]\) we can quickly find the consecutive occurrences of \(P_1\) and \(P_2\) with distance in \({[}\alpha , \beta ]\), i.e., pairs of subsequent occurrences with distance within the range. We present data structures that use linear space and query time \({\widetilde{O}}(|P_1|+|P_2|+n^{2/3})\) for existence and counting and \({\widetilde{O}}(|P_1|+|P_2|+n^{2/3}\hbox {occ}^{1/3})\) for reporting. We complement this with a conditional lower bound based on the set intersection problem showing that any solution using \({\widetilde{O}}(n)\) space must use \({\widetilde{\Omega }}(|P_1| + |P_2| + \sqrt{n})\) query time. To obtain our results we develop new techniques and ideas of independent interest including a new suffix tree decomposition and hardness of a variant of the set intersection problem.

We present a new multithreaded implementation for the computationally demanding direct volume rendering (DVR) of volumetric data sets on desktop multicore processors using ray casting. The new implementation achieves interactive rendering of very large volumes, even on high resolution screens. Our implementation is based on a new algorithm that combines an object‐order traversal of the volumetric data followed by a focused ray casting. Using a very compact data structure, our method starts with a quick association of data subcubes with fine‐grain screen tiles appearing along the viewing direction in front‐to‐back order. The next stage uses very limited ray casting on the generated sets of subcubes while skipping empty or transparent space and applying early ray termination in an effective way. Our multithreaded implementation makes use of new dynamic techniques to ensure effective memory management and load balancing. Our software enables a user to interactively explore large data sets through DVR while arbitrarily specifying a 2D transfer function. We test our system on a wide variety of well‐known volumetric data sets on a two‐processor Clovertown platform, each consisting of a Quad‐Core 1.86 GHz Intel Xeon Processor. Our experimental tests demonstrate DVR at interactive rates for the largest data sets that can fit in the main memory on our platform. These tests also indicate a high degree of scalability, excellent load balancing, and efficient memory management across the data sets used. Copyright © 2009 John Wiley &amp; Sons, Ltd.

Efficient pebbling for list traversal synopses with application to program rollback