Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

Neural networks offer various possibilities for function approximation. When provided a set of data points, the network learns to approximate the underlying function that generates those points. Although the network can be very efficient, the amount computation needed during the learning process can be very high. In order to improve this process, we explore the parallelization for the random scanning of starting points selected for the gradient descent algorithm using Cell-BE multiprocessor environment. We show the application of this method for approximating 3D nonlinear function, as well as for predicting 2D time series. We show that the parallel tracing of gradient descent trajectories of the 3D function approximation allows identifying a suitable starting condition for implementing an efficient gradient descent, while being able deliver the required accuracy of approximation in a shorter time. In 2D time series prediction the attained advantage is the possibility to achieve simultaneous prediction for various numbers of steps ahead. It is shown how the Cell-BE multiprocessor offers a convenient parallel environment for the above solutions.

Inspired by Polyak’s heavy-ball method, this paper proposes an adaptive deterministic block coordinate descent method with momentum (mADBCD) for efficiently solving large-scale linear least-squares problems. The proposed method introduces a novel column selection criterion based on the Euclidean norm of the residual vector of the normal equation. In contrast to classical block coordinate descent methods, mADBCD does not require a fixed pre-partitioning of the column indices of the coefficient matrix and avoids the expensive computation of Moore Penrose pseudoinverses of submatrices at each iteration. The method adaptively updates the block index set at each step, thereby improving both flexibility and scalability. When the coefficient matrix is of full column rank, we prove that mADBCD converges linearly to the unique solution of the least-squares problem. Numerical experiments are conducted to show that mADBCD outperforms several recent block coordinate descent methods in terms of iteration count and CPU time. In particular, when solving extremely sparse least-squares problems, mADBCD is the first block coordinate descent method reported to achieve CPU time nearly comparable to that of the classical least squares QR (LSQR) method [Paige and Saunders, ACM Trans. Math. Softw., 8 (1982)].

The effectiveness of sparse matrix techniques for directly solving large-scale linear least-squares problems is severely limited if the system matrix A has one or more nearly dense rows. In this paper, we partition the rows of A into sparse rows and dense rows (As and Ad) and apply the Schur complement approach. A potential difficulty is that the reduced normal matrix AsTAs is often rank-deficient, even if A is of full rank. To overcome this, we propose explicitly removing null columns of As and then employing a regularization parameter and using the resulting Cholesky factors as a preconditioner for an iterative solver applied to the symmetric indefinite reduced augmented system. We consider complete factorizations as well as incomplete Cholesky factorizations of the shifted reduced normal matrix. Numerical experiments are performed on a range of large least-squares problems arising from practical applications. These demonstrate the effectiveness of the proposed approach when combined with either a sparse parallel direct solver or a robust incomplete Cholesky factorization algorithm.

In reverse engineering, the point cloud data processing is the key, it will directly affect the quality and efficiency of the subsequent reconstruction of the surface. However, the amount of data obtained by three-dimensional laser scanning is often larger and redundant, which will increase the difficulty of the surface reconstruction, therefore, the point cloud data is needed to streamline. Depth analysis of the efficiency and accuracy of curvature grid method, a secondary streamlined algorithm is proposed, which take use of non-uniform grid streamline law based on octree to streamline to the point cloud, and re-use curvature streamline method based on B-spline surface fitting to streamline it. This method, whose streamline ratio is relatively high, can effectively retain the feature point cloud data, providing a great convenience for the follow-up of reverse engineering.

Surface display techniques, including contour mapping, usually use an interpolant when data values are on a rectangular grid; but when values are irregularly located some form of assumed or estimated distance-weighting function is often used to estimate the influence of data points at other locations. If moving average (or some other) techniques are then applied, slopes may be zero at data points; extrema may fall at data points exclusively; or surfaces may not pass through data points.An alternative method is proposed whereby a map is automatically divided into suitable triangular domains with a data point at each vertex. Surface estimation and contour plotting are then performed independently for each triangle using a measured or best-fit plane associated with each data point. The approach requires the utilization of three non-original aspects: 1) the derivation of a local homogeneous area coordinate system for any arbitrary triangle; 2) the construction of a data-structure linking each triangular domain with its three neighbouring triangles and three associated data points, and 3) the use of a conforming triangular finite-element interpolating function.Use of the first two of these concepts permits the economic generation and optimization of a triangular mesh from a set of data points. Optimization criteria used to define the best triangular partition are described in some detail, along with computer timing for this step. Use of the first and third concepts permits the interpolation of a smooth surface over the whole map even though each triangular element is estimated and plotted independently. The requirements for a suitable interpolating function are discussed, and an interpolant is suggested that preserves elevation and slope at each data point as well as elevation and slope continuity between domains. Extrema need not be located at data points but are constrained by their associated planes. Contour line segments are produced by division of each triangle into N2 sub-triangles, elevation estimation at each resulting node, and linear interpolation and plotting withing each sub-triangle. Resolution is then a function of N and local data point density. Extension of the method to higher dimensions is briefly discussed.

The solution of the unconstrained weighted linear least-squares problem is known to be a convex combination of the basic solutions formed by the nonsingular subsystems if the weight matrix is diagonal and positive definite. In particular, this implies that the norm of this solution is uniformly bounded for any diagonal and positive definite weight matrix. In addition, the solution set is known to be the relative interior of a finite set of polytopes if the weight matrix varies over the set of positive definite diagonal matrices. In this paper, these results are reviewed and generalized to the set of weight matrices that are symmetric, positive semidefinite, and diagonally dominant and that give unique solution to the least-squares problem. This is done by means of a particular symmetric diagonal decomposition of the weight matrix, giving a finite number of diagonally weighted problems but in a space of higher dimension. Extensions to equality-constrained weighted linear least-squares problems are given. A discussion of why the boundedness properties do not hold for general symmetric positive definite weight matrices is given. The motivation for this research is from interior methods for optimization.

Preconditioned GMRES methods with incomplete Givens orthogonalization method for large sparse least-squares problems

Avoiding out-of-distribution (OOD) data is critical for training supervised machine learning models in the medical imaging domain. Furthermore, obtaining labeled medical data is difficult and expensive since it requires expert annotators like doctors, radiologists, etc. Active learning (AL) is a well-known method to mitigate labeling costs by selecting the most diverse or uncertain samples. However, current AL methods do not work well in the medical imaging domain with OOD data. We propose Diagnose (avoiDing out-of-dIstribution dAta usinG submodular iNfOrmation meaSurEs), an active learning framework that can jointly model similarity and dissimilarity, which is crucial in mining in-distribution data and avoiding OOD data at the same time. Particularly, we use a small number of data points as exemplars that represent a query set of in-distribution data points and another set of exemplars that represent a private set of OOD data points. We illustrate the generalizability of our framework by evaluating it on a wide variety of real-world OOD scenarios. Our experiments verify the superiority of Diagnose over the state-of-the-art AL methods across multiple domains of medical imaging.

Nowadays, the influence of machine learning algorithms on our lives is booming gradually, and gradient descent (GD) method is a typical big data analysis method that can be applied to various machine learning algorithms. Traditional GD method has its advantages in dealing with the optimal parameter finding issue for partial loss functions, but the optimal parameter finding process may be trapped into local minimum sometimes if the loss function with certain convexity; moreover, traditional GD cannot meet the requirement of high accuracy. Fractional order GD, which is the core of fixed memory step GD, has better performance by comparing with traditional GD, but still have some problems in the aspect of convergence. On the other hand, t-distributed machine learning algorithm for dimensionality reduction, can reduce the dimensionality of data in high-dimensional space to two-dimensional or three-dimensional space, and can convert the original data into more comprehensible images. In order to solve the problem mentioned above reasonably, in this paper, a fixed-memory step GD method combined with t-distributed stochastic neighbor embedding (t-SNE) is employed to replace the traditional GD method, so as to keep the consistency of low-dimensional data being mapped from high-dimensional data, and to achieve efficient data visualization. The effectiveness of proposed t-SNE algorithm based on fixed memory step gradient descent method has been verified finally via numerical experiment.

Curve and surface representation by iterative B-spine fit to a data point set.

This chapter deals with the problem of fitting curves to data points, a classical optimization problem in Computer-Aided Geometric Design (CAGD). This issue plays an important role in real problems such as construction of car bodies, ship hulls, airplane fuselage, and other free-form objects. A typical example comes from reverse engineering where free-form shapes are extracted from clouds of scanned data points. In this chapter we address this issue by applying a powerful bio-inspired method called Artificial Immune Systems (AIS). The AIS can be understood as a computational methodology based upon metaphors of the biological immune system. As such, there is not one but several AIS algorithms. In this chapter we focus on the clonal selection algorithm (CSA), which explicitly takes into account the affinity maturation of the immune response. This algorithm is applied to fit Bezier curves to given sets of data points. Some illustrative examples show the good performance of our approach.

Wilkinson defined a sparse matrix as one with enough zeros that it pays to take advantage of them. 1 This informal yet practical definition captures the essence of the goal of direct methods for solving sparse matrix problems. They exploit the sparsity of a matrix to solve problems economically: much faster and using far less memory than if all the entries of a matrix were stored and took part in explicit computations. These methods form the backbone of a wide range of problems in computational science. A glimpse of the breadth of applications relying on sparse solvers can be seen in the origins of matrices in published matrix benchmark collections (Duff and Reid 1979 a , Duff, Grimes and Lewis 1989 a , Davis and Hu 2011). The goal of this survey article is to impart a working knowledge of the underlying theory and practice of sparse direct methods for solving linear systems and least-squares problems, and to provide an overview of the algorithms, data structures, and software available to solve these problems, so that the reader can both understand the methods and know how best to use them.

Problem statement: A direct method, such as least squares technique is usually used to solve problems involving matching a curve or a surface to a set of data points. The solution obtained by this direct method is precise or very good in approximation, but computationally not very efficient. Thus, in this study, we propose an indirect approach using Particle Swarm Optimization (PSO) technique as an alternative. Approach: As a case study, we use conic curve which satisfy C0 continuity to be fitted to a given set of data points. PSO, a soft computing method is employed to optimize the control points and weights which are then used in conic equations. Results: Best fitted conic curve that represents all the given data points is then obtained. Conclusion: We use an indirect technique of soft computing methods, i.e., PSO to fit a curve to a given data set. We believe that other types of soft computing based heuristic procedures may also be used to solve related problems or to find its effectiveness.

Field-scale investigation for a period of more than four months was conducted to evaluate the performance of a landfill for biogas extraction upon the injection of food waste leachate (FWL), a liquid organic waste generated from the food waste recycling facilities in Korea. The target was set at recovering about 50–60 % methane from the landfill gas (LFG) at extraction rates varying between 10 and 30 m3/h. An application of the artificial neural network (ANN) was presented in this paper to predict the performance parameter namely methane percentage (%). The input parameters to the network were LFG extraction rate (m3/h) and landfill leachate: FWL ratio, respectively, which were obtained from the field-scale investigation. Four different back error propagation learning algorithms were used to train the ANN for a comparative analysis, and the best among them was selected. To substantiate our claim, performance of the network was analyzed for different set of training and test data points. Predictions were attained by appropriately selecting the network parameters and, adequately training the network with 130 set of data points. The accuracy of back propagation neural network (BPNN)-based model predictions was evaluated by calculating the correlation coefficient (R) and mean absolute percentage error values. The results from this predictive modeling work showed that BPNNs were able to predict the methane percentage of the LFG in an acceptable range.

We provide experimental evidence that current desktop computers feature enough computational power to solve large-scale dense linear algebra problems. While the high computational cost of the numerical methods for solving these problems can be tackled by the multiple cores of current processors, we propose to use the disk to store the large data structures associated with these applications. Our results also show that the limited amount of RAM and the comparatively slow disk of the system pose no problem for the solution of very large dense linear systems and linear least-squares problems. Thus, current desktop computers are revealed as an appealing, cost-effective platform for research groups that have to deal with large dense linear algebra problems but have no direct access to large computing facilities.