The algorithm for identical object searching with bounded worst-case complexity and linear memory

Supervised low rank indefinite kernel approximation using minimum enclosing balls

As describe an efficient hierarchical method to compute attention in the Transformer architecture. The proposed attention mechanism exploits a matrix structure similar to the Hierarchical Matrix (H-Matrix) developed by the numerical analysis community, and has linear run time and memory complexity. We perform extensive experiments to show that the inductive bias embodied by our hierarchical attention is effective in capturing the hierarchical structure in the sequences typical for natural language and vision tasks. Our method is superior to alternative sub-quadratic proposals by over +6 points on average on the Long Range Arena benchmark. It also sets a new SOTA test perplexity on One-Billion Word dataset with 5x fewer model parameters than that of the previous-best Transformer-based models.

We introduced inference and search algorithm separately, because they seem to have useful but complementary properties. We saw that if we apply search over an AND/OR space and if, in addition, we use context-based caching, search has the same worst-case complexity as inference, both are time and space exponentially bounded by the treewidth. However, when the treewidth is too high, the needed memory is not available and schemes that are more flexible in trading space for time are needed. This is one of the main virtues of search compared with inference. Search can be done in linear memory and accommodates space for time tradeoff quite flexibly. In fact, we have already observed that the class of AND/OR search AO(i) of searching the AND/OR context minimal search space and caching tables based on bounded (by i) context only, can be trade time and space effectively. In this chapter we discuss several additional approaches trading space for time that are based on combining search with inference. We demonstrate the principles of such hybrids in the context of OR tree search, first, and then extend it to AND/OR search.

As an important part in network management system, rapid and accurate anomaly detection is one of the preconditions to guarantee the effective work of the network. In general, the network management system will monitor the data of multiple indicators, and the manifestations of network anomalies are complex in the data. Most existing anomaly detection approaches have encountered some trouble in network anomaly detection because of the periodicity of time series data or high time complexity and memory cost working on high-dimensional data. Aiming at the deficiency of present methods of network anomaly detection, we propose an adapted method based on iForest algorithm. We construct some feature extractors through statistical methods to highlight different abnormal behaviors in different indicator and then use the extracted feature data for the construction and prediction of iForest. Combining specific feature extractors, we can eliminate the effects of periodicity, or specify the detection of peaks or troughs to adapt to different indicators. By means of iForest algorithm which has linear time complexity and low memory requirement, our method makes a rapid detection with large dataset and works well in high dimensional network management data.

The abundance of time series data in various domains and their high dimensionality characteristic are challenging for harvesting useful information from them. To tackle storage and processing challenges, compression-based techniques have been proposed. Our previous work, Domain Series Corpus (DSCo), compresses time series into symbolic strings and takes advantage of language modeling techniques to extract from the training set knowledge about different classes. However, this approach was flawed in practice due to its excessive memory usage and the need for a priori knowledge about the dataset. In this paper we propose DSCo-NG, which reduces DSCo’s complexity and offers an efficient (linear time complexity and low memory footprint), accurate (performance comparable to approaches working on uncompressed data) and generic (so that it can be applied to various domains) approach for time series classification. Our confidence is backed with extensive experimental evaluation against publicly accessible datasets, which also offers insights on when DSCo-NG can be a better choice than others.

Nuclei segmentation and classification provide an essential basis for tumor immune microenvironment analysis. The previous nuclei segmentation and classification models require splitting large images into smaller patches for training, leading to two significant issues. First, nuclei at the borders of adjacent patches often misalign during inference. Second, this patch-based approach significantly increases the model's training and inference time. Recently, Mamba has garnered attention for its ability to model large-scale images with linear time complexity and low memory consumption. It offers a promising solution for training nuclei segmentation and classification models on full-sized images. However, the Mamba orientation-based scanning method lacks account for category-specific features, resulting in suboptimal performance in scenarios with imbalanced class distributions. To address these challenges, this paper introduces a novel scanning strategy based on category probability sorting, which independently ranks and scans features for each category according to confidence from high to low. This approach enhances the feature representation of uncertain samples and mitigates the issues caused by imbalanced distributions. Extensive experiments conducted on four public datasets demonstrate that our method outperforms state-of-the-art approaches, delivering superior performance in nuclei segmentation and classification tasks.

The probabilistic classification vector machine is a very effective and generic probabilistic and sparse classifier. A recently published incremental version improved the runtime complexity to quadratic costs. We derive the Nystrom approximation for asymmetric matrices to obtain linear runtime and memory complexity for the incremental probabilistic classification vector machine while keeping similar prediction performance.

SparseHC: A Memory-efficient Online Hierarchical Clustering Algorithm

In order to solve the problems of large memory consumption and complex model of CNN in traffic signs recognition, we propose an improved deep learning method DCR (Dual Compression Recognition) based on BNN (Binary Neural Network) model. We compress BNN's network structure and dataset (remove the color information on the dataset). Experiments show that the DCR model with only one binary convolutional layer is enough to achieve about 75.18% accuracy on the GTSRB dataset, while the trainable parameters are only 1.4M and the model memory is only 16.1M. Moreover, the DCR model with three binary convolutional layers has an accuracy comparable to that of the classical CNN's models, with an accuracy of 92.86%, while the trainable parameters is only 0.83M and the model memory is only 9.60M. As a result, most arithmetic operations have been simplified, and the memory access and memory size have been reduced by 32 times. In addition, because the color information of the dataset is removed, the training speed of the model is effectively improved. All these make our model more suitable for embedded deployment with lower cost.

This paper deals with the spatialization of multiple compressed audio streams. A new approach is proposed based on the combination of subband-domain filtering methods with linear decomposition methods of HRTFs filters, and more generally the filters used by sound spatialization techniques (binaural, transaural, ambisonic, etc). Such a combination allows both computation complexity reduction and memory size saving for sound spatialization systems of multiple compressed audio signals. These advantages are very useful for any implementation on limited capabilities terminals (computation power and memory size) which is the case of mobile and portable devices.

Dynamic 3-sided planar range queries with expected doubly-logarithmic time

Tensor completion is the task of completing multi-aspect data represented as a tensor by accurately predicting missing entries in the tensor. It is mainly solved by tensor factorization methods, and among them, Tucker factorization has attracted considerable interests due to its powerful ability to learn latent factors and even their interactions. Although several Tucker methods have been developed to reduce the memory and computational complexity, the state-of-the-art method still 1) generates redundant computations and 2) cannot factorize a large tensor that exceeds the size of memory. This paper proposes FTcom, a fast and scalable Tucker factorization method for tensor completion. FTcom performs element-wise updates for factor matrices based on coordinate descent, and adopts a novel caching algorithm which stores frequently-required intermediate data. It also uses a tensor file for disk-based data processing and loads only a small part of the tensor at a time into the memory. Experimental results show that FTcom is much faster and more scalable compared to all other competitors. It significantly shortens the training time of Tucker factorization, especially on real-world tensors, and it can be executed on a billion-scale tensor which is bigger than the memory capacity within a single machine.

Nonnegative matrix factorization (NMF) often relies on the separability condition for tractable algorithm design. Separability-based NMF is mainly handled by two types of approaches, namely, greedy pursuit and convex programming. A notable convex NMF formulation is the so-called self-dictionary multiple measurement vectors (SD-MMV), which can work without knowing the matrix rank a priori, and is arguably more resilient to error propagation relative to greedy pursuit. However, convex SD-MMV renders a large memory cost that scales quadratically with the problem size. This memory challenge has been around for a decade, and a major obstacle for applying convex SD-MMV to big data analytics. This work proposes a memory-efficient algorithm for convex SD-MMV. Our algorithm capitalizes on the special update rules of a classic algorithm from the 1950s, namely, the Frank-Wolfe (FW) algorithm. It is shown that, under reasonable conditions, the FW algorithm solves the noisy SD-MMV problem with a memory cost that grows linearly with the amount of data. To handle noisier scenarios, a smoothed group sparsity regularizer is proposed to improve robustness while maintaining the low memory footprint with guarantees. The proposed approach presents the first linear memory complexity algorithmic framework for convex SD-MMV based NMF. The method is tested over a couple of unsupervised learning tasks, i.e., text mining and community detection, to showcase its effectiveness and memory efficiency.

RWKV achieves Transformer-level performance with linear memory complexity for long sequences. However, its inherent sequentiality limits hardware parallelism and triggers memory-bound bottlenecks via frequent off-chip weight accesses. To address these challenges, we propose HFRWKV, an FPGA-based hardware accelerator for RWKV. Within the matrix operation module, we propose a novel Δ-PoT hardware-friendly hybrid-precision quantization strategy, which enhances performance while maintaining acceptable accuracy. For the complex operations including exponentiation and division, we introduce a method featuring reusable architectures combined with lookup tables or piecewise linear approximation, which is algorithmically refined to effectively balance precision and hardware resource consumption. We adopt a fully on-chip computing system integrating parallel matrix-vector array and efficient pipelining, employing computation reordering and chunked double buffering to eliminate data bottlenecks and maximize throughput. We implement HFRWKV on the Alveo U50 and U280 platform. Experimental results show that compared to a CPU, a throughput improvement of 63.48× and an energy efficiency improvement of 139.17×. Compared to GPUs, achieves a throughput improvement of 32.33× and an energy efficiency improvement of 171.36×.