On the mathematical foundations of learning

Nelson Goodman’s new riddle of induction forcefully illustrates a challenge that must be confronted by any adequate theory of inductive inference: provide some basis for choosing among alternative hypotheses that fit past data but make divergent predictions. One response to this challenge is to distinguish among alternatives by means of some epistemically significant characteristic beyond fit with the data. Statistical learning theory takes this approach by showing how a concept similar to Popper’s notion of degrees of testability is linked to minimizing expected predictive error. In contrast, formal learning theory appeals to Ockham’s razor, which it justifies by reference to the goal of enhancing efficient convergence to the truth. In this essay, I show that, despite their differences, statistical and formal learning theory yield precisely the same result for a class of inductive problems that I call strongly VC ordered, of which Goodman’s riddle is just one example.

The best algorithm for a computational problem generally depends on the “relevant inputs,” a concept that depends on the application domain and often defies formal articulation. While there is a large body of literature on empirical approaches to selecting the best algorithm for a given application domain, there has been surprisingly little theoretical analysis of the problem. This paper adapts concepts from statistical and online learning theory to reason about application-specific algorithm selection. Our models capture several state-of-the-art empirical and theoretical approaches to the problem, ranging from self-improving algorithms to empirical performance models, and our results identify conditions under which these approaches are guaranteed to perform well. We present one framework that models algorithm selection as a statistical learning problem, and our work here shows that dimension notions from statistical learning theory, historically used to measure the complexity of classes of binary- and re...

The present paper shows how statistical learning theory and machine learning models can be used to enhance understanding of AI-related epistemological issues regarding inductive reasoning and reliability of generalisations. Towards this aim, the paper proceeds as follows. First, it expounds Price’s dual image of representation in terms of the notions of e-representations and i-representations that constitute subject naturalism. For Price, this is not a strictly anti-representationalist position but rather a dualist one (e- and i-representations). Second, the paper links this debate with machine learning in terms of statistical learning theory becoming more viable epistemological tool when it abandons the perspective of object naturalism. The paper then argues that machine learning grounds a form of knowing that can be understood in terms of e- and i-representation learning. Third, this synthesis shows a way of analysing inductive reasoning in terms of reliability of generalisations stemming from a structure of e- and i-representations. In the age of Artificial Intelligence, connecting Price’s dual view of representation with Deep Learning provides an epistemological way forward and even perhaps an approach to how knowing is possible.

Data is available like never before. We believed that back in the 1990s, but corpora are even larger today than they were then, and corpora will continue to grow for some time to come. Thus far, corpus sizes have been limited by our ability to collect data, but we are rapidly approaching a fundamental limit on supply of written and spoken language. There are only so many people in the world, and they have only so much time to communicate with one another. It is becoming feasible to digitize a non-trivial fraction of the world’s communication. This ability is creating new opportunities for new audiences to join in on the fun. Google Ngrams makes it easy for anyone to apply corpus-based methods to half a trillion words (4% of all books ever printed). The popular press is referring to corpus methods and Google Ngrams as “addictive.” Computer Scientists are talking about “digital immortality” (recording much of human communication and storing it forever). Digital immortality may not be a reality just yet, but psychologists are currently recording most of what children say and hear between 2 months and 2 years of age in order to better understand language acquisition. As the world becomes digitized, there will be many applications of corpus-based methods that include lexicography (and so much more).

Recent research has shown the need to present knowledge to students in a manner consistent with the characteristics of the human brain, especially during learning mathematics, which contributes to the development of achievement and inductive reasoning. The current study aimed to develop achievement and inductive reasoning by employing brain-compatible learning theory in teaching and learning mathematics for sixth-grade students in Oman. The sample included (74) students, who were divided into two equal groups: experimental and control. Study data were collected using mathematical achievement tests and inductive reasoning. There were significant differences between the two groups in the post-application of the achievement and inductive reasoning tests in favor of the experimental group, which indicates that teaching mathematics according to the theory of learning compatible with the brain had a significant impact on the development of achievement and inductive reasoning among the participants. The study recommended the need to develop and teach mathematics curricula in a way that allows for the practice of many activities and exercises that contribute to the development of students’ achievement and inductive reasoning skills.

Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear;in particular, how different architectures may lead to inductive bias when trained using gradient-based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias; in particular, transitioning between kernel and feature learning regimes.

Learning and evolving are two basics for data mining. As compared with classical learning theory based on objective function with minimizing training errors, the recently evolutionary computing has had an efficient approach for constructing optimal model without the minimizing training errors. The global search of evolutionary computing in solution space can settle the local optima problems of learning models. In this research, combining co-evolving algorithm into statistical learning theory, we propose an co-evolutionary computing for statistical learning theory for overcoming local optima problems of statistical learning theory. We apply proposed model to classification and prediction problems of the learning. In the experimental results, we verify the improved performance of our model using the data sets from UCI machine learning repository and KDD Cup 2000.

In statistical learning theory (SLT), the key theorem and the bounds on the rate of uniform convergence of learning processes provide theoretical basis for the applied research of support vector machine etc., so they play important roles in SLT. In the study of two aspects, samples which we deal with are supposed to be noise-free. But it is not always the case because of the influence of human or environmental factors. With a view of this, we propose and prove the key theorem and discuss the bounds on the rate of uniform convergence of learning processes when samples are corrupted by noise

The rise and fall of artificial neural networks is well documented in the scientific literature of both computer science and computational chemistry. Yet almost two decades later, we are now seeing a resurgence of interest in deep learning, a machine learning algorithm based on multilayer neural networks. Within the last few years, we have seen the transformative impact of deep learning in many domains, particularly in speech recognition and computer vision, to the extent that the majority of expert practitioners in those field are now regularly eschewing prior established models in favor of deep learning models. In this review, we provide an introductory overview into the theory of deep neural networks and their unique properties that distinguish them from traditional machine learning algorithms used in cheminformatics. By providing an overview of the variety of emerging applications of deep neural networks, we highlight its ubiquity and broad applicability to a wide range of challenges in the field, including quantitative structure activity relationship, virtual screening, protein structure prediction, quantum chemistry, materials design, and property prediction. In reviewing the performance of deep neural networks, we observed a consistent outperformance against non-neural networks state-of-the-art models across disparate research topics, and deep neural network-based models often exceeded the "glass ceiling" expectations of their respective tasks. Coupled with the maturity of GPU-accelerated computing for training deep neural networks and the exponential growth of chemical data on which to train these networks on, we anticipate that deep learning algorithms will be a valuable tool for computational chemistry. © 2017 Wiley Periodicals, Inc.

A method for modernizing the route of technical document flow at an enterprise manufacturing high-tech products proposed using elements of the theory of artificial neural networks. The traditional route of the processes of approval of technical documentation in the environment of an automated enterprise system revised. With analysis using elements of the theory of artificial neural networks, elements of the route that require improvement identified. An improved model of the technical document flow route in the environment of an enterprise automated system is proposed, which made it possible to improve the quality of the processes of approval of the enterprise's technical documentation.

Deep learning has transformed various sectors, introducing new applications and opportunities. However, the underlying physical mechanisms or mathematical theories responsible for its success remain fundamental questions. This inquiry explores the connection between deep learning algorithms and established scientific principles with the aim of uncovering the mysteries behind their remarkable capabilities. By bridging the gap between deep learning, neural networks, and scientific knowledge, we can develop robust and interpretable models with enhanced capabilities. This ongoing research involves collaboration across diverse fields to unveil the hidden intricacies of deep learning algorithms and their links to physical phenomena. The ultimate goal is to contribute to the potential of the journal by examining the theory, design and application of neural networks and machine learning, focusing on the effectiveness of neural network paradigms for deep learning and their connections to physical events. By examining the intersection of deep learning, neural networks, and physical phenomena, we aim to advance our understanding and use of neural networks and machine learning in many areas of space, pushing the boundaries of excellence in science and engineering

An experimental study on classification of thyroid histopathology images using transfer learning

<title>Abstract</title> Brain morphology emerges from the interplay of genetic programming and mechanical forces. However, its fractal-like folding patterns, regional anisotropy, and uneven curvature distributions pose longstanding challenges for quantitative analysis in medical and developmental studies. These difficulties are exacerbated by the limited availability of labeled data for existing statistical learning approaches. Here, we introduce a theory-grounded physics-transfer (PT) learning framework that enables generalization analysis and prediction in complex physical systems by leveraging the continuity of underlying physical laws across different levels of complexity. Specifically, mechanistic insights into nonlinear elasticity of simple, analytically tractable geometries are embedded directly into neural networks and then transferred to brain models through a sequence of physics-anchored domains, providing, for the first time, a formulation that integrates explicit physics constraints into learning theory. In this sense, PT advances classical statistical learning and domain adaptation theories by enabling generalization analysis in regimes where purely statistical assumptions are insufficient. Building on PT, we derive a generalization bound that explains the model’s strong performance in feature characterization and morphogenesis prediction for brain. The results reveal that localized deformation modes, rather than global geometry, dominantly shape cortical folding dynamics. Beyond predictive accuracy, the framework also yields reduced-dimensional evolutionary representations that distill the essential physics of the highly folded cerebral cortex. Validation against medical imaging data and expert clinical assessment demonstrates the promise of physics-aware digital-twin technologies for understanding, diagnosing, and ultimately intervening in the morphological complexity of the developing and diseased brain.

Statistical physics is the area of physics which studies the properties of systems composed of many microscopic particles (like atoms and molecules). When combined, the interactions between these particles produce the macroscopic features of the systems. The systems are usually characterized by a very large number of variables and the limited possibilities for observing the properties of the components of the system. For these reasons, solving problems arisen in Statistical Physics with analytical approaches is usually ineffective and sometimes impossible. However, statistical approaches (such as Monte Carlo simulation) can provide acceptable approximations for solutions of these problems. Moreover, recent studies showed that nature inspired metaheuristics (like Genetic Algorithms, Evolutionary Strategies, Particle Swarm Optimization, etc) can also be used to simulate, analyse, and optimize such systems, providing fast and accurate results. Apart from physical implications, problems from Statistical Physics are also important in fields like biology, chemistry, mathematics, communications, economy, sociology, etc. We will present two important problems from Statistical Physics and discuss how one can use Particle Swarm Optimization (PSO) to tackle them. First, we will discuss how the realvalued version of PSO can be used to mini mize the energy of a system composed of repulsive point charges confined on a sphere. This is known as the Thomson problem and it is included in Stephen Smale's famous list of 18 unsolved mathematical problems to be solved in the 21st century. This problem also arises in biology, chemistry, communications, economy, etc. Latter on, we will discuss how the binary version of PSO can be used to search ground states of Ising spin glasses. Spin glasses are materials that simultaneously present ferromagnetic and anti-ferromagnetic interactions among their atoms. A ground state of a spin glass is a configuration of the system in which this has the lowest energy possible. Besides its importance for Statistical Physics, this problem has applications in neural network theory, computer science, biology, etc.

Computer science is increasingly becoming an essential 21st century skill. As school systems around the world recognize the importance of computer science, demand for teachers who have the knowledge and skills to deliver computer science instruction is rapidly growing. Yet a number of recent studies indicate that teachers report low confidence and limited understanding of computer science, frequently confusing basic computer literacy skills with computer science. This is true for both teachers at the K-8 level as well as secondary education teachers who frequently transition to computer science from other content areas, such as mathematics. As computer science is not yet included in most teacher preparation programs, professional development is a critical step in efforts to prepare in-service teachers to deliver high-quality computer science instruction. To date, however, research on best practices in computer science professional development has been severely lacking in the literature, making it difficult for researchers and practitioners alike to examine effective in-service preparation models.This book provide examples of professional development approaches that help teachers integrate aspects of computing in existing curricula at the K-8 level or deliver stand-alone computer science courses at the secondary school level. Further, this book identifies computational competencies for teachers, promising pedagogical strategies that advance teacher learning, as well as alternative pathways for ongoing learning including microcredentials. The primary audience of the book is graduate students and faculty in educational technology, educational or cognitive psychology, learning theory, curriculum and instruction, computer science, instructional systems and learning sciences. Additionally, the book will serve as a valuable addition to education practitioners and curriculum developers as well as policy makers looking to increase the number of teachers who are prepared to deliver computing education.