Adaptive model selection using empirical complexities

We consider complexity penalization methods for model selection. These methods aim to choose a model to optimally trade off estimation and approximation errors by minimizing the sum of an empirical risk term and a complexity penalty. It is well known that if we use a bound on the maximal deviation between empirical and true risks as a complexity penalty, then the risk of our choice is no more than the approximation error plus twice the complexity penalty. There are many cases, however, where complexity penalties like this give loose upper bounds on the estimation error. In particular, if we choose a function from a suitably simple convex function class with a strictly convex loss function, then the estimation error (the difference between the risk of the empirical risk minimizer and the minimal risk in the class) approaches zero at a faster rate than the maximal deviation between empirical and true risks. In this paper, we address the question of whether it is possible to design a complexity penalized model selection method for these situations. We show that, provided the sequence of models is ordered by inclusion, in these cases we can use tight upper bounds on estimation error as a complexity penalty. Surprisingly, this is the case even in situations when the difference between the empirical risk and true risk (and indeed the error of any estimate of the approximation error) decreases much more slowly than the complexity penalty. We give an oracle inequality showing that the resulting model selection method chooses a function with risk no more than the approximation error plus a constant times the complexity penalty.We gratefully acknowledge the support of the NSF under award DMS-0434383. Thanks also to three anonymous reviewers for useful comments that improved the presentation.

With the help of 5G network, edge intelligence (EI) can not only provide distributed, low-latency, and high-reliable intelligent services, but also enable intelligent maintenance and management of smart city. However, the constantly changing available computing resources of end devices and edge servers cannot continuously guarantee the performance of intelligent inference. In order to guarantee the sustainability of intelligent services in smart city, we propose the Adaptive Model Selection and Partition Mechanism (AMSPM) in 5G smart city where EI provides services, which mainly consists of Adaptive Model Selection (AMS) and Adaptive Model Partition (AMP). In AMSPM, the model selection and partition of deep neural network (DNN) are formulated as an optimization problem. Firstly, we propose a recursive-based algorithm named AMS based on the computing resources of edge devices to derive an appropriate DNN model that satisfies the latency demand of intelligent services. Then, we adaptively partition the selected DNN model according to the computing resources of edge devices. The experimental results demonstrate that, when compared with state-of-the-art model selection and partition mechanisms, AMSPM not only reduces latency but also enhances computing resource utilization.

In this paper, we apply semi-Lagrangian discontinuous Galerkin (SLDG) methods for linear hyperbolic equations in one space dimension and analyze the error between the numerical and exact solutions under the L 2 L^2 -norm. In all the previous works, the theoretical analysis of the SLDG method would suggest a suboptimal convergence rate due to the error accumulation over time steps. However, numerical experiments demonstrate an optimal convergence rate and, if the terminal time is large, a superconvergence rate. In this paper, we will prove optimal convergence and optimal superconvergence rates. There are three main difficulties: 1. The error analysis on overlapping meshes. Due to the nature of the semi-Lagrangian time discretization, we need to introduce the background Eulerian mesh and the shifted mesh. The two meshes are staggered, and it is not easy to construct local projections and to handle the error accumulation during time evolution. 2. The superconvergence of time-dependent terms under the L 2 L^2 -norm. The error of the numerical and exact solutions can be divided into two parts, the projection error and the time-dependent superconvergence term. The projection strongly depends on the superconvergence rates. Therefore, we need to construct a sequence of projections and gradually improve the superconvergence rates. 3. The stopping criterion of the sequence of projections. The sequence of projections are basically of the same form. We need to show that the projections exist up to some certain order since the superconvergence rate cannot be infinity. Hence, we will seek some “hidden” condition for the existence of the projections. In this paper, we will solve all the three difficulties and construct several local projections to prove the optimal convergence and superconvergence rates. Numerical experiments verify the theoretical findings.

In this paper we address the problem of dense depth map estimation from sparse noisy range data to reconstruct large heterogeneous outdoor scenes. We propose a surface inpainting solution through energy minimisation with an adaptive selection of surface regularisers among a set of well known convex and non-convex regularisers. In fact, the selection of norm is pivotal with respect to the intrinsic surface characteristics. Our goal is to show how dense interpolation of sparse range data can be leveraged of more exotic and non-convex regularisers such as the log and logTGV [1] which can better capture the scene geometry. In contrast to state of the art solutions, we do not restrict ourselves to this set of norms, instead we search for the most apt norm for each semantically segmented part of the scene. Our energy model selection use Bayesian optimisation to learn the best choice of free parameters. This results in an adaptive model selection and the generalisation of well studied regularisation norms. We conclude with a detailed experimental analysis of our approach using a basis of four norms over a set of challenging outdoor scenes.

A prediction algorithm for time series based on adaptive model selection

Model selection: A Lagrange optimization approach

Most model selection procedures use a fixed penalty penalizing an increase in the size of a model. These nonadaptive selection procedures perform well only in one type of situation. For instance, Bayesian information criterion (BIC) with a large penalty performs well for “small” models and poorly for “large” models, and Akaike's information criterion (AIC) does just the opposite. This article proposes an adaptive model selection procedure that uses a data-adaptive complexity penalty based on a concept of generalized degrees of freedom. The proposed procedure, combining the benefit of a class of nonadaptive procedures, approximates the best performance of this class of procedures across a variety of different situations. This class includes many well-known procedures, such as AIC, BIC, Mallows's Cp, and risk inflation criterion (RIC). The proposed procedure is applied to wavelet thresholding in nonparametric regression and variable selection in least squares regression. Simulation results and an asymptotic analysis support the effectiveness of the proposed procedure.

We consider the wavelet-based estimators of mean regression function with long memory moving average errors and investigate their asymptotic rates of convergence based on thresholding of empirical wavelet coefficients. We show that these estimators achieve nearly optimal minimax convergence rates within a logarithmic term over a large range of Besov function classes \(B^{s}_{p,q}\). Therefore, in the presence of long memory non-Gaussian moving average noise, wavelet estimators still achieve nearly optimal convergence rates and provide explicitly the extraordinary local adaptability. The theory is illustrated with some numerical examples.

Budget constraints arise in many computer vision problems. Computational costs limit many automated recognition systems while crowdsourced systems are hindered by monetary costs. We leverage wide variability in image complexity and learn adaptive model selection policies. Our learnt policy maximizes performance under average budget constraints by selecting “cheap” models for low complexity instances and utilizing descriptive models only for complex ones. During training, we assume access to a set of models that utilize features of different costs and types. We consider a binary tree architecture where each leaf corresponds to a different model. Internal decision nodes adaptively guide model-selection process along paths on a tree. The learning problem can be posed as an empirical risk minimization over training data with a non-convex objective function. Using hinge loss surrogates we show that adaptive model selection reduces to a linear program thus realizing substantial computational efficiencies and guaranteed convergence properties.Keywordstest-time budgetadaptive model selectioncost-sensitive learning

Nested simulation is typically used to estimate the functional of a conditional expectation. Considerable research has been performed on point estimation for various functionals. However, the quantification of the statistical uncertainty in the point estimator, for instance, using confidence intervals (CIs), has not been extensively investigated. In this article, we establish central limit theorems with the asymptotically optimal convergence rate of for nested simulation with different forms of functionals, where denotes the total computational effort. Based on these theorems, we develop a unified CI framework that can ensure that both the mean squared error of the point estimator and CI width attain the optimal convergence rate. Numerical examples are presented, and the results are found to be consistent with the theoretical results. Experimental results demonstrate that the proposed framework outperforms the existing methods for CI construction in terms of the CI widths and convergence rates.

The convergence of stochastic interacting particle systems in the mean-field limit to solutions of conservative stochastic partial differential equations is established, with optimal rate of convergence. As a second main result, a quantitative central limit theorem for such SPDEs is derived, again, with optimal rate of convergence. The results apply, in particular, to the convergence in the mean-field scaling of stochastic gradient descent dynamics in overparametrized, shallow neural networks to solutions of SPDEs. It is shown that the inclusion of fluctuations in the limiting SPDE improves the rate of convergence, and retains information about the fluctuations of stochastic gradient descent in the continuum limit.

SUMMARY We consider estimation of a variance function g in regression problems. Such estimation requires simultaneous estimation of the mean function f. We obtain clear results on the extent to which the smoothness of f influences best rates of convergence for estimating g. For example, in nonparametric regression with two derivatives on g, ‘classical' rates of convergence are possible if and only if the unknown f satisfies a Lipschitz condition of order 13 or more. If a parametric model is known for g, then g may be estimated n1/2 consistently if and only if f is Lipschitz of order 12 or more. Optimal rates of convergence are attained by Kernel estimators.

This paper presents an accurate distress classification method via adaptive Bayesian network model selection for maintenance inspection of road structures. The main contribution of this paper is adaptive selection of two Bayesian network models concerning classification performance. The proposed method trains a tag-based Bayesian network model based on inspection items and estimates its classification performance. Furthermore, for distresses that degrade the classification performance of the tag-based Bayesian network model, the proposed method trains another multifeature Bayesian network model based on inspection items and distress images. Consequently, the proposed method can adaptively select optimal Bayesian network models according to the estimated performance of the tag-based Bayesian network model. In actual maintenance inspection, distresses are generally classified either from inspection items alone or from both inspection items and visual information of distress images—i.e., distress classification has two patterns. Therefore the adaptive model selection approach is suitable for this classification scheme. Experimental results show that the proposed method outperforms several comparative methods and is suitable for actual maintenance inspection due to its low computation costs.

The recently proposed String Gradient Weighted Moving Finite Element (SGWMFE) method is extended to include remeshing and refining. The method simultaneously determines, at each time step, the solution of the governing partial differential equations and an optimal location of the finite element nodes. It has previously been applied to the nonlinear time-dependent two-dimensional shallow water equations, under the demanding conditions of large Coriolis forces, inducing large mesh and field rotation. Such effects are of major importance in geophysical fluid dynamics applications. Two deficiencies of the original SGWMFE method are (1) possible tangling of the mesh which causes the method's failure, and (2) no mechanism for global refinement when necessary due to the constant number of degrees of freedom. Here the method is extended in order to continue computing solutions when the meshes become too distorted, which happens quickly when the flow is rotationally dominant. Optimal rates of convergence are obtained when remeshing is applied. The method is also extended to include refinement to enable handling of new physical phenomena of a smaller scale which may appear during the solution process. It is shown that the errors in time are kept under control when refinement is necessary. Results of the extended method for some example problems of water hump release are presented.

Real-time machine vision applications running on resource-constrained embedded systems face challenges for maintaining performance. An especially challenging scenario arises when multiple applications execute at the same time, creating contention for the computational resources of the system. This contention results in increase in inference delay of the machine vision applications, which can be unacceptable for time-critical tasks. To address this challenge, we propose an adaptive model selection framework that mitigates the impact of system contention and prevents unexpected increases in inference delay by trading off the application accuracy minimally. The framework has two parts, which are performed pre-deployment and at runtime. The pre-deployment part profiles the system for contention in a black-box manner and produces a model set that is specifically optimized for the contention levels observed in the system. The runtime part predicts the inference delays of each model considering the system contention and selects the best model according to the predictions for each frame. Compared to a fixed individual model with similar accuracy, our framework improves the performance by significantly reducing the inference delay violations against a specified threshold. We implement our framework on the Nvidia Jetson TX2 platform and show that our approach achieves greater than 20% reductions in delay violations over the individual baseline models.