On the adaptive deterministic block Kaczmarz method with momentum for solving large-scale consistent linear systems

Inspired by Polyak’s heavy-ball method, this paper proposes an adaptive deterministic block coordinate descent method with momentum (mADBCD) for efficiently solving large-scale linear least-squares problems. The proposed method introduces a novel column selection criterion based on the Euclidean norm of the residual vector of the normal equation. In contrast to classical block coordinate descent methods, mADBCD does not require a fixed pre-partitioning of the column indices of the coefficient matrix and avoids the expensive computation of Moore Penrose pseudoinverses of submatrices at each iteration. The method adaptively updates the block index set at each step, thereby improving both flexibility and scalability. When the coefficient matrix is of full column rank, we prove that mADBCD converges linearly to the unique solution of the least-squares problem. Numerical experiments are conducted to show that mADBCD outperforms several recent block coordinate descent methods in terms of iteration count and CPU time. In particular, when solving extremely sparse least-squares problems, mADBCD is the first block coordinate descent method reported to achieve CPU time nearly comparable to that of the classical least squares QR (LSQR) method [Paige and Saunders, ACM Trans. Math. Softw., 8 (1982)].

Due to its simplicity and efficiency, the first-order gradient method has been extensively employed in training neural networks. Although the optimization problem of the neural network is non-convex, recent research has proved that the first-order method is capable of attaining a global minimum during training over-parameterized neural networks, where the number of parameters is significantly larger than that of training instances. Momentum methods, including the heavy ball (HB) method and Nesterov's accelerated gradient (NAG) method, are the workhorse of first-order gradient methods owning to their accelerated convergence. In practice, NAG often exhibits superior performance than HB. However, current theoretical works fail to distinguish their convergence difference in training neural networks. To fill this gap, we consider the training problem of the two-layer ReLU neural network under over-parameterization and random initialization. Leveraging high-resolution dynamical systems and neural tangent kernel (NTK) theory, our result not only establishes tighter upper bounds of the convergence rate for both HB and NAG, but also provides the first theoretical guarantee for the acceleration of NAG over HB in training neural networks. Finally, we validate our theoretical results on three benchmark datasets.

Sensitivity and uncertainty analysis is becoming increasingly widespread in many fields of engineering and sciences, encompassing practically all of the experimental data-processing activities and many computational modeling and process simulation activities. There are many methods, based either on deterministic or statistical concepts, for performing sensitivity and uncertainty analysis. However, a precise, unified terminology across all methods does not seem to exist, yet often, identical words (e.g., “sensitivity”) may not necessarily describe identical quantities, particularly when stemming from conceptually distinct (statistical versus deterministic) methods. Furthermore, the relative strengths and weaknesses of the various methods do not seem to have been reviewed comparatively in the literature published thus far.This paper is the first part of a comparative review, written in two parts, that focuses on the salient features of the statistical and deterministic methods currently used for local and global sensitivity and uncertainty analysis of both large-scale computational models and indirect experimental measurements. Deterministic methods are analyzed in Part I, while statistical methods are highlighted in Part II.Part I of this review commences by highlighting the deterministic methods for computing local sensitivities, namely, the so-called Brute-Force Method (based on recalculations), the Direct Method (including the Decoupled Direct Method), the Green’s Function Method, the Forward Sensitivity Analysis Procedure (FSAP), and the Adjoint Sensitivity Analysis Procedure (ASAP). Except for the Brute-Force Method, it is emphasized that local sensitivities can be computed exactly and exhaustively only by using deterministic methods. Furthermore, it is noted that the Direct Method and the FSAP require at least as many model evaluations as there are parameters, while the ASAP requires a single model evaluation of an appropriate adjoint model whose source term is related to the response under investigation. If this adjoint model is developed simultaneously with the original model, then the adjoint model requires relatively modest additional resources to develop and implement. If, however, the adjoint model is constructed a posteriori, considerable skills may be required for its successful development and implementation. Nevertheless, the ASAP is the most efficient method to use for computing local sensitivities of large-scale systems, where the number of parameters, and parameter variations, exceeds the number of responses of interest.The Global ASAP (GASAP) is also highlighted as it appears to be the only deterministic method published thus far for performing genuinely global analysis of nonlinear systems. The GASAP uses both the forward and the adjoint sensitivity systems to explore, exhaustively and efficiently, the entire phase-space of system parameters and dependent variables in order to obtain complete information about the important global features of the physical system, namely, the critical points of the response and the bifurcation branches and/or turning points of the system’s state variables.

The momentum method has become the workhorse in the deep learning community. To theoretical understand its success, researchers put efforts in demystifying its convergence properties when optimizing neural networks. For the convex problem, it is well-known that the triple momentum (TM) method owns the fastest theoretical convergence rate among all first-order methods. However, there exists no theoretical convergence results about the TM method in solving the non-convex neural networks training problem, let alone its acceleration guarantee. In this paper, we focus on the training process of the TM method for a two-layer ReLU neural network. Inspired by the accurate characterization of the high-resolution dynamical system, we consider the high-resolution ordinary differential equation (ODE) of the TM method. Under the over-parameterized assumption, we derive that the original non-convex optimization problem can be transformed to a strongly convex task. By applying an appropriate Lyapunov function, we prove that the TM method can linearly converge to a global minimum. Compared to the heavy ball method and Nesterov’s accelerated gradient method, our result provides the first guarantee for the acceleration of the TM method in training neural networks. Through empirical experiments, the accelerated convergence of the TM method and the effect of over-parameterization are validated.

This chapter treats the theory of ordinary differential equations, both linear and nonlinear.Sections 1–4 establish existence and uniqueness theorems for ordinary differential equations. The first section gives some examples of first-order equations, mostly nonlinear, to illustrate certain kinds of behavior of solutions. The second section shows, in the presence of continuity for a vectorvalued F satisfying a “Lipschitz condition,” that the first-order system y′ = F(t, y) has a unique local solution satisfying an initial condition y (t0) = y0. Since higher-order equations can always be reduced to first-order systems, these results address existence and uniqueness for nth-order equations as a special case. Section 3 shows that the solutions to a system depend well on the initial condition and on any parameters that are present in F. Section 4 applies these results to existence of integral curves for a vector field and to construction of coordinate systems from families of integral curves.Sections 5–8 concern linear systems. Section 5 shows that local solutions of linear systems may be extended to global solutions and that in the homogeneous case the vector space of global solutions has dimension equal to the size of the system. The method of variation of parameters reduces the solution of any linear system to the solution of a homogeneous linear system. Sections 6–7 identify explicit solutions to nth-order linear equations and first-order linear systems. The “Jordan canonical form” of a square matrix plays a role in the case of a system. Section 8 discusses power-series solutions to second-order homogeneous linear equations whose coefficients are given by convergent power series, as well as solutions that arise in the case of regular singular points. Two kinds of special functions are mentioned that result from this study-Legendre polynomials and Bessel functions.KeywordsOrdinary Differential EquationUniqueness TheoremDiagonal EntryLipschitz ConditionIntegral CurfThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The DFOM method is an iterative method for computing the Drazin-inverse solution of consistent or inconsistent linear systems of the form Ax = b, where A ϵ ℂ n × n is a singular and in general non-Hermitian matrix that has an arbitrary index. This method is generally used with restarting. But the restarting often slows down the convergence and DFOM often stagnates. We show that adding some approximate error vectors or approximate eigenvectors (corresponding to a few of the smallest eigenvalues) to the Krylov subspace can improve the convergence just like the method proposed by R. Morgan in [8]. We derive the implementation of these methods and present some numerical examples to show the advantages of these methods.

A decentralized adaptive control method is proposed for large-scale systems with unknown time-delayed nonlinear interconnections unmatched in control inputs. It is assumed that the time-delayed interaction terms are bounded by unknown nonlinear bounding functions. The nonlinear bounding functions and uncertain nonlinear functions of large-scale systems are compensated by the function approximation technique using neural networks. The dynamic surface control method is extended to design the proposed memoryless local controller for each subsystem of uncertain nonlinear large-scale time delay systems. Therefore, although the interconnected systems consist of a large number of subsystems, the proposed controller can be designed simply. We prove that all the signals in the total closed-loop system are semiglobally uniformly bounded and the control errors converge to an adjustable neighborhood of the origin. Finally, an example is given to demonstrate the effectiveness and applicability of the proposed scheme.

The randomized Kaczmarz (RK) method is an iterative method for approximating the least-squares solution of large linear systems of equations. The standard RK method uses sequential updates, making parallel computation difficult. Here, we study a parallel version of RK where a weighted average of independent updates is used. We analyze the convergence of RK with averaging and demonstrate its performance empirically. We show that as the number of threads increases, the rate of convergence improves and the convergence horizon for inconsistent systems decreases.

The process of refactoring enhances software quality by modifying its design composition while preserving its core framework. However, addressing code smells without appropriate prioritization can be ineffective. Code smells significantly increase maintenance costs and obstruct system evolution. Refactoring sequencing techniques mitigate these issues by improving a system’s internal structure without altering its external behavior. In large-scale systems, the sheer number of code smells can be overwhelming, and not all can be automatically resolved. Hence, prioritizing code smells based on criteria such as risk and importance is essential. This paper introduces a novel hybrid approach utilizing the hybrid spotted hyena and emperor penguin (HSHEP) optimization-based algorithm. This approach aims to optimize the sequence of code smell bugs by incorporating maintainer opinions and requirements, thereby maximizing the resolution of critical code smells. Unlike existing technologies, the HSHEP algorithm combines the strengths of two optimization strategies, offering a unique and innovative solution to refactoring challenges. To validate the effectiveness of the proposed method, it was applied to various large-scale open-source systems, analyzing five different types of code smells. Results demonstrated a significant improvement in maintenance efficiency and system evolution, confirming the superior performance and practical applicability of the HSHEP-based approach. Received: 22 April 2024 | Revised: 13 June 2024 | Accepted: 30 June 2024 Conflicts of Interest The authors declare that they have no conflicts of interest in this work. Data Availability Statement Data sharing is not applicable to this article as no new data were created or analyzed in this study. Author Contribution Statement Ritika Maini: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization, Funding acquisition. Navdeep Kaur: Supervision. Amandeep Kaur: Project administration.

This paper studies fractional differential equations (FDEs) with mixed fractional derivatives. Existence, uniqueness, stability, and asymptotic results are derived.

The accurate evaluation of large-scale wireless communication systems necessitates efficient channel emulation and modeling techniques.While deterministic methods, such as ray-tracing (RT) simulation, offer high accuracy, their computational complexity poses challenges in dynamic environments involving a moving receiver (Rx).To address this, a grid-based channel emulation technique is employed, where the simulation site is divided into grids, and deterministic channel parameters are precomputed at each node.The channel impulse response (CIR) is then synthesized through interpolation of these parameters.However, as path parameters are estimated independently at each node, an effective path association algorithm is required.This work proposes a clustering-based approach that utilizes angulardomain parameters to associate paths.Then, path clustering is performed prior to interpolation to satisfy the hardware constraints of the emulator.The algorithm is validated using an RT-simulated dataset representing a smart office environment at a carrier frequency ( f C ) of 28.35 GHz.

This study proposes two improved gradient descent parameter estimation algorithms for rational state-space models with time-delay. These two algorithms, based on intelligent search method and momentum method, can simultaneously estimate the time-delay and parameters without the matrix eigenvalue calculation in each iteration. Compared with the traditional gradient descent algorithm, the improved algorithms come with two advantages: having quicker convergence rates and less computational efforts, particularly meaningful for those large-scale systems. A simulated example is selected to illustrate the efficiency of the proposed algorithms.

Surface Stabilization of O3-type Layered Oxide Cathode to Protect the Anode of Sodium Ion Batteries for Superior Lifespan.

A performance model of non-deterministic particle transport on large-scale systems

The analysis of errors which arise in the numerical solution of linear algebraic systems is considered, and a simple method using matrix and vector norms is suggested as being suitable for presentation to engineering students in a first course on numerical linear algebra. An argument for computation of the inverse matrix is presented, in which the mathematical problem is considered as part of the engineering design problem.