Distributed-Memory Sparse Kernels for Machine Learning

We present four numerical methods for computing the singular value decomposition (SVD) of large sparse matrices on a multiprocessor architecture. We emphasize Lanczos and subspace iteration-based methods for determining several of the largest singular triplets (singular values and corresponding left- and right-singular vectors) for sparse matrices arising from two practical applications: information retrieval and seismic reflection tomography. The target architectures for our implementations are the CRAY-2S/4–128 and Alliant FX/80. The sparse SVD problem is well motivated by recent information-retrieval techniques in which dominant singular values and their corresponding singular vectors of large sparse term-document matrices are desired, and by nonlinear inverse problems from seismic tomography applications which require approximate pseudo-inverses of large sparse Jacobian matrices. This research may help advance the development of future out-of-core sparse SVD methods, which can be used, for example, to handle extremely large sparse matrices 0 × (106) rows or columns associated with extremely large databases in query-based information-retrieval applications.

Introduction: In order to assess how well conventional and hybrid pitch detection techniques perform in speech processing applications, a comparative analysis of the two types of methods is conducted. Method: A proposed hybrid approach, Proposed PEF+CEP, is examined alongside five traditional algorithms, namely Normalized Correlation Function (NCF), Pitch Estimation Filter (PEF), Log- Harmonic Summation (LHS), Summation of Residual Harmonics (SRH) and Cepstrum Pitch Determination (CEP). The effectiveness is evaluated using performance metrics like accuracy, specificity, sensitivity, and Gross Pitch Error (GPE). Results and Discussion: Our findings show that the accuracy and specificity of the traditional methods are impressive; the accuracy and sensitivity of the suggested hybrid method surpass their performance, with an astounding 98.8% accuracy and 99.2% sensitivity. Conclusion: Furthermore, the Proposed PEF+CEP method is a promising solution for accurate and dependable pitch detection in speech processing applications because it strikes a strong balance between computational efficiency and robustness. These results open up new avenues for research in the field of speech processing and demonstrate the potential of hybrid approaches.

A systematic comparison of runtime load balancing algorithms for massively parallel rigid particle dynamics

Domain gaps are significant obstacles to the clinical implementation of machine learning (ML) solutions for medical image analysis. Although current research emphasizes new training methods and network architectures, the specific impact of prevalence shifts on algorithms in real-world applications is often overlooked. Differences in class frequencies between development and deployment data are crucial, particularly for the widespread adoption of artificial intelligence (AI), as disease prevalence can vary greatly across different times and locations. Our contribution is threefold. Based on a diverse set of 30 medical classification tasks (1) we demonstrate that lack of prevalence shift handling can have severe consequences on the quality of calibration, decision threshold, and performance assessment. Furthermore, (2) we show that prevalences can be accurately and reliably estimated in a data-driven manner. Finally, (3) we propose a new workflow for prevalence-aware image classification that uses estimated deployment prevalences to adjust a trained classifier to a new environment, without requiring additional annotated deployment data. Comprehensive experiments indicate that our proposed approach could contribute to generating better classifier decisions and more reliable performance estimates compared to current practice.

Federated Learning (FL) is an emerging learning paradigm that enables the collaborative learning of different nodes without ex-posing the raw data. However, a critical challenge faced by the current federated learning algorithms in real-world applications is the long-tailed data distribution, i.e., in both local and global views, the numbers of classes are often highly imbalanced. This would lead to poor model accuracy on some rare but vital classes, e.g., those related to safety in health and autonomous driving applications. In this paper, we propose BalanceFL, a federated learning frame-work that can robustly learn both common and rare classes from a long-tailed real-world dataset, addressing both the global and local data imbalance at the same time. Specifically, instead of letting nodes upload a class-drifted model trained on imbalanced private data, we design a novel local update scheme that rectifies the class imbalance, forcing the local model to behave as if it were trained on ideal uniform distributed data. To evaluate the performance of BalanceFL, we first adapt two public datasets to the long-tailed federated learning setting, and then collect a real-life IMU dataset for action recognition, which includes more than 10,000 data sam-ples and naturally exhibits the global long tail effect and the local imbalance. On all of these three datasets, BalanceFL outperforms state-of-the-art federated learning approaches by a large margin.

Multi-view clustering has become a widely studied problem in the area of unsupervised learning. It aims to integrate multiple views by taking advantages of the consensus and complimentary information from multiple views. Most of the existing works in multi-view clustering utilize the vector-based representation for features in each view. However, in many real-world applications, instances are represented by graphs, where those vector-based models cannot fully capture the structure of the graphs from each view. To solve this problem, in this paper we propose a Multi-view Clustering framework on graph instances with Graph Embedding (MCGE). Specifically, we model the multi-view graph data as tensors and apply tensor factorization to learn the multi-view graph embeddings, thereby capturing the local structure of graphs. We build an iterative framework by incorporating multi-view graph embedding into the multi-view clustering task on graph instances, jointly performing multi-view clustering and multi-view graph embedding simultaneously. The multi-view clustering results are used for refining the multi-view graph embedding, and the updated multi-view graph embedding results further improve the multi-view clustering. Extensive experiments on two real brain network datasets (i.e., HIV and Bipolar) demonstrate the superior performance of the proposed MCGE approach in multi-view connectome analysis for clinical investigation and application.

Parallel direct method for linear equations

A scalable approach computes in operationally-compatible time the energy dispatch under uncertainty for electrical power grid systems of realistic size with thousands of scenarios. The authors propose several algorithmic and implementation advances in their parallel solver PIPS-IPM for stochastic optimization problems. New developments include a novel, incomplete, augmented, multicore, sparse factorization implemented within the PARDISO linear solver and new multicore- and GPU-based dense matrix implementations. They show improvement on the interprocess communication on Cray XK7 and XC30 systems. PIPS-IPM is used to solve 24-hour horizon power grid problems with up to 1.95 billion decision variables and 1.94 billion constraints on Cray XK7 and Cray XC30, with observed parallel efficiencies and solution times within an operationally defined time interval. To the authors' knowledge, "real-time"-compatible performance on a broad range of architectures for this class of problems hasn't been possible prior to this work.

With the wide adoption of AI applications, there is a pressing need of\nenabling real-time neural network (NN) inference on small embedded devices, but\ndeploying NNs and achieving high performance of NN inference on these small\ndevices is challenging due to their extremely weak capabilities. Although NN\npartitioning and offloading can contribute to such deployment, they are\nincapable of minimizing the local costs at embedded devices. Instead, we\nsuggest to address this challenge via agile NN offloading, which migrates the\nrequired computations in NN offloading from online inference to offline\nlearning. In this paper, we present AgileNN, a new NN offloading technique that\nachieves real-time NN inference on weak embedded devices by leveraging\neXplainable AI techniques, so as to explicitly enforce feature sparsity during\nthe training phase and minimize the online computation and communication costs.\nExperiment results show that AgileNN's inference latency is >6x lower than the\nexisting schemes, ensuring that sensory data on embedded devices can be timely\nconsumed. It also reduces the local device's resource consumption by >8x,\nwithout impairing the inference accuracy.\n

We describe a procedure for determining a few of the largest singular values of a large sparse matrix. The method by Golub and Kent which uses the method of modified moments for estimating the eigenvalues of operators used in iterative methods for the solution of linear systems of equations is appropriately modified in order to generate a sequence of bidiagonal matrices whose singular values approximate those of the original sparse matrix. A simple Lanczos recursion is proposed for determining the corresponding left and right singular vectors. The potential asynchronous computation of the bidiagonal matrices using modified moments with the iterations of an adapted Chebyshev semi-iterative (CSI) method is an attractive feature for parallel computers. Comparisons in efficiency and accuracy with an appropriate Lanczos algorithm (with selective re-orthogonalization) are presented on large sparse (rectangular) matrices arising from applications such as information retrieval and seismic reflection tomography. This procedure is essentially motivated by the theory of moments and Gauss quadrature.

Boundary element methods are being successfully used for modeling parasitic effects in cutting-edge circuit design. The dense system matrix generated therein presents a time and memory bottleneck. Fast iterative solver techniques, developed to address the problem, suffer from convergence issues which become pronounced for large number of right hand sides as is the case for massively coupled systems. In this paper an iteration free solution scheme is presented. The dense matrix is rendered sparse by applying multilevel multipole expansions, and the resultant sparse matrix is solved by a traditional sparse matrix solver. The accuracy and time and memory requirements for the solver are compared against the regular methods. The advantage of the presented method over the corresponding iterative scheme is also demonstrated.

Reliable Internet of Things (IoT) service discovery is a significant task in intelligent service-oriented IoT systems. Collaborative filtering (CF) turns out to be an effective solution to IoT service discovery. However, the traditional CF framework is facing the following challenges: inefficiency in learning the high-order interactions between users and IoT services and ineffectively making use of geographical location information. Moreover, deploying a CF framework in real-world distributed IoT systems poses another significant challenge in terms of reliability assurance and privacy protection. For bridging this gap, we propose an innovative integrated collaborative filtering framework (ICF) with incorporating the location-aware quality of IoT services into a heterogeneous graph embedding model. Meanwhile, a Geohash-based privacy-preservation mechanism is introduced to encoding the location information into a short string for protecting the sensitive location information. The proposed ICF framework is an integrated architecture combining advanced graph embedding learning and heterogeneous side information. And a joint objective optimization function is designed in the graph embedding learning to qualify automatic IoT service features. In this way, the hidden user-service interaction information and location-aware quality semantic features can be explored in an effective and reliable way. Furthermore, the learned location-aware quality embedding vector is incorporated to discover the reliable services among all IoT services with the mini-batch online clustering-based CF algorithm. The experimental findings illustrate the significant efficacy and reliability of our proposed ICF framework in large scale IoT service discovery scenarios.

The concepts of efficiency and effectiveness must be addressed in conducting research into using a Evolutionary Algorithm (EA) for optimization problems. The increased use of evolutionary approaches for real-world applications, containing multiple objectives and high dimensionality, has led to the design and generation of a number of Multiobjective Evolutionary Algorithms (MOEA). When analyzing these algorithms, the issues of effectiveness and efficiency are extremely important and typically drive the urge to parallelize these algorithms. The parallelization of MOEAs is a relatively new concept, with few researchers contributing work in this area. This parallelization process is not a simple task and involves the analysis of various parallel models and the parameters associated with these models. This paper presents a thorough analysis of the various parallel MOEA models, the issues associated with these models and recommendations for using these models in MOEAs. In particular, these parallelization concepts are applied to the Multiobjective Messy Genetic Algorithm II.

Sparse-Dense Matrix Multiplication (SpMM) on GPUs has gained significant attention because of its importance in modern applications and the increasing computing power of GPUs in the last decade. Previous SpMM studies have focused on the importance of storage format and load balance for the overall performance of SpMM on GPUs. However, very little attention has been paid to the efficacy of coalesced memory access in improving the efficiency of data loading, which incurs a notable overhead that amounts to an average of more than 32% of the overall performance, according to our experimental observation. Existing state-of-the-art (SOTA) solutions fail to adequately support coalesced memory access of both sparse and dense matrices between the global memory and threads on GPUs. In this paper, we propose an efficient algorithm called Swift.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>Swift is available at https://github.com/MinttHu/Swift.git that speeds up the loading of both sparse and dense matrices of SpMM on modern GPUs. Leveraging coalesced memory access, Swift achieves high loading efficiency by sorting both the columns of the sparse matrix and elements of the dense matrix based on the number of non-zero elements and balancing the load by handling the regular and irregular parts differently and judiciously. Swift takes the Compressed Sparse Column format as an implementation case study to prove the concept and gain insights. We conduct a comprehensive comparison of Swift with four SOTA solutions: ASpT, cuSPARSE, RoDe, and Sputnik, using the full SuiteSparse Matrix Collection as the workload. The experimental results on RTX 4080s, RTX 3090Ti, A100, and V100 demonstrate that our method outperforms the baselines significantly.

Matrix factorization is one of the most popular techniques for prediction problems in the fields of intelligent systems and data mining. It has shown its effectiveness in many real-world applications such as recommender systems. As a collaborative filtering method, it gives users recommendations based on their previous preferences or ratings. Due to the extreme sparseness of the ratings matrix, active learning is used for eliciting ratings for a user to get better recommendations. In this paper, we propose a new matrix factorization model called Enhanced SVD ESVD which combines the classic matrix factorization method with a specific rating elicitation strategy. We evaluate the proposed ESVD method on the Movielens data set, and the experimental results suggest its effectiveness in terms of both accuracy and efficiency, when compared with traditional matrix factorization methods and active learning methods.