Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers

BackgroundThe new EU Medical Device Regulation (MDR) places greater importance on the role of clinical evidence to establish safety and performance. Article 54 of the MDR calls for expert committees to independently review the scientific, technical, and clinical evidence supporting the market authorization of certain novel devices independently from the established process of Notified Body reviews. These experts provide a review and opinion that ultimately is taken into consideration alongside the information reviewed by the Notified Body during the review process. Four expert committees (General and Plastic Surgery and Dentistry; Orthopaedics, Traumatology, Rehabilitation, Rheumatology; Circulatory System; and Neurology) have published at least one Scientific Opinion (SO) under the Clinical Evaluation Consultation Procedure (CECP) in 2021–2022.MethodsThe four expert committees with published CECP opinions were reviewed to assess the academic backgrounds and professional expertise of each member with respect to clinical, technical, and biological domains on a 0–2 scale for each domain. A content review was conducted on the 10 CECP opinions published by these committees to assess their consistency with the goals and outcome expectations set by the MDR. The extent of content related to each of the clinical, technical, and biological domains was also assessed on a 0–2 scale.ResultsAll committees were composed primarily by members with strong clinical expertise, but only a few had strong technical and biological expertise. Across committees, the average scores of members related to academic background and professional expertise both ranged from 1.64 to 2.00 in the clinical domain, but only 0–0.15 and 0.15–0.69, respectively, in the biological domain, and 0.12–0.55 and 0.23–0.73, respectively, in the technical domain. A content review for the 10 SOs showed that all opinions focused exclusively or primarily on the clinical evidence. Three contained a modest amount of additional text directed at technical/engineering issues and five at biological issues.ConclusionExpert committees are composed predominantly of expert clinical reviewers but have many fewer members with significant technical or biological expertise. This may limit the ability of the committees to evaluate the significant technical and biological risks that are often best understood by preclinical testing. Broadening the expertise across the committees may improve the depth of their benefit/risk critiques.

Much information sources model imprecision by the way of unimodal, consonant and continuous probability density functions (pdfs). We consider here in the framework of belief functions on real numbers, agents of evidence deduced from such pdfs. First are singletons plausibilities in conjunctive and disjunctive combinations proposed to basically merge agents of evidence with consonant focal elements. Second are partial and global conflict calculation methods provided. An application shows the plausibility curves and conflict values obtained in case of combination operations done on Gaussian based agents and at last, an example of conflict management based on an RCR-S adaptive rule of combination is given.

Probability estimation and information principles

Methods of ensemble postprocessing in which continuous probability density functions are constructed from ensemble forecasts by centering functions around each of the ensemble members have come to be called Bayesian model averaging (BMA) or “dressing” methods. Here idealized ensemble forecasting experiments are used to show that these methods are liable to produce systematically unreliable probability forecasts of climatologically extreme weather. It is argued that the failure of these methods is linked to an assumption that the distribution of truth given the forecast can be sampled by adding stochastic perturbations to state estimates, even when these state estimates have a realistic climate. It is shown that this assumption is incorrect, and it is argued that such dressing techniques better describe the likelihood distribution of historical ensemble-mean forecasts given the truth for certain values of the truth. This paradigm shift leads to an approach that incorporates prior climatological information into BMA ensemble postprocessing through Bayes’s theorem. This new approach is shown to cure BMA’s ill treatment of extreme weather by providing a posterior BMA distribution whose probabilistic forecasts are reliable for both extreme and nonextreme weather forecasts.

Abstract. Volcanic eruptions are among the most jeopardizing natural events due to their potential impacts on life, assets, and the environment. In particular, atmospheric dispersal of volcanic tephra and aerosols during explosive eruptions poses a serious threat to life and has significant consequences for infrastructures and global aviation safety. The volcanic island of Jan Mayen, located in the North Atlantic under trans-continental air traffic routes, is considered the northernmost active volcanic area in the world with at least five eruptive periods recorded during the last 200 years. However, quantitative hazard assessments on the possible consequences for the air traffic of a future ash-forming eruption at Jan Mayen are nonexistent. This study presents the first comprehensive long-term volcanic hazard assessment for the volcanic island of Jan Mayen in terms of ash dispersal and concentration at different flight levels. In order to delve into the characterization and modeling of that potential impact, a probabilistic approach based on merging a large number of numerical simulations is adopted, varying the volcano's eruption source parameters (ESPs) and meteorological scenario. Each ESP value is randomly sampled following a continuous probability density function (PDF) based on the Jan Mayen geological record. Over 20 years of meteorological data is considered in order to explore the natural variability associated with weather conditions and is used to run thousands of simulations of the ash dispersal model FALL3D on a 2 km resolution grid. The simulated scenarios are combined to produce probability maps of airborne ash concentration, arrival time, and persistence of unfavorable conditions at flight levels 50 and 250 (FL050 and FL250). The resulting maps can serve as an aid during the development of civil protection strategies, to decision-makers and aviation stakeholders, in assessing and preventing the potential impact of a future ash-rich eruption at Jan Mayen.

In this paper, the influence of uncertain model parameters on vehicle lateral dynamics steady states is studied. These steady states are widely used in driver assistant systems and enable highly automated driving functions. A methology for propagating random variables through the single-track vehicle model is presented. The random variables are described using continuous probability density functions. It is shown, that the influence of parameter uncertanties scales differently to different steady states. The effect on the path curvature and the lateral acceleration is highlighted. An example for a fault detection scheme is given and analysed in context of the given uncertainty. The faults that can be detected without violating a given true-positive rate and a given false-positive rate are calculated. The method can be used to quickly identify possible problems arising from uncertain parameters.

Predicting community noise levels requires the addition of background ambient sound levels and the source’s contribution to ambient sound levels. Baseline data are obtained, from community sound surveys, in the form of histograms and cumulative distributions. Source (for example, a construction site) sound level data are also obtained as histograms and cumulative distributions. The problem of combining the two distributions is made somewhat difficult by the logarithmic nature of the physical variable. The prediction is accomplished by the mathematical convolution of the variables representing baseline and the source. Since the data are provided as histograms rather than as continuous probability density functions, it is convenient to decompose the density functions into a sequence of Dirac delta functions. The result can be programmed for a rapid and efficient computation. Results are provided for the combination of various construction site sound level distributions (e.g., normal and bimodal) with community background ambient sound. Data are shown of the effect on community noise of a source contribution which is attenuated by distance. Subject Classification: [43]50.75.

Due to the influence of weather, light, season and angle changes, the appearance of objects changes visually, which makes it difficult for unmanned vehicles that rely on visual positioning to complete their positioning work. This paper proposes a coordinated positioning method that is composed of semantic information and geometric relationships distribution (GRD), which improves the robustness of unmanned vehicle location under the above conditions. First, we improved the FAST-SCNN semantic segmentation network and replaced its fully connected layer with the conv4-3 module to prevent the spatial information of the image from being lost in the fully connected layer. At the same time, the conv4-3 layer contains the richest semantic information, we use image semantic content to create a dense and prominent scene description. These prominent descriptions were learned from a large data set of perceptual changes. The method can accurately segment geometrically stable image regions. We combine the characteristics of these highlighted areas with the existing overall representation to produce a more robust scene descriptor. Second, a method is designed to integrate the matching of semantic labels and geometric distribution relations, which is a new closed loop location recognition label and landmark map. The geometric pair relationship between the ground marks is encoded as a continuous probability density function, the GRD function, which is expressed by a Laguerre polynomial and Fourier series basis expansion. This orthogonal basis representation allows for efficient computation of rotation and translation invariants, which are used to compare signatures and search for potential loop closure candidates. Finally, we evaluate our method with some of the most advanced algorithms, such as OpenSeqSLAM, AlexNet and VSO, to demonstrate its advantages. The experimental results for representative data sets show that the method based on Fast-SCNN is superior to other methods.

We present a hybrid speech recognition system for speaker independent continuous speech recognition. The system combines a novel information theory based neural network (NN) paradigm and discrete Hidden Markov models (HMMs) including state-of-the-art techniques like state clustered triphones. The novel NN type is trained by an algorithm based on principles of self-organization that achieves maximum mutual information between the generated output labels and the basic phonetic classes. The structure of the hybrid system is quite similar to a classical VQ-HMM system but the vector quantizer (VQ) is replaced by the NN. To evaluate the system we use the speaker independent part of the resource management (RM) database. We obtained an important improvement by introducing a novel kind of context dependent basic classes used by the acoustic processor. The average RM recognition result with a word-pair grammar is now 95.2% what is significantly better than a classical VQ-system, slightly better than a different hybrid system with a recurrent network as probability estimator, and very close to the best continuous probability density function (PDF) HMM speech recognizers.

Face recognition has become an important topic within the field of pattern recognition and computer vision. In this field a number of different approaches to feature extraction, modeling and classification techniques have been tested. However, many questions concerning the optimal modeling techniques for high performance face recognition are still open. The face recognition system developed by our research group uses a discrete cosine transform (DCT) combined with the use of pseudo-2D hidden Markov models (P2DHMM). In the past our system used continuous probability density functions and was tested on a smaller database. This paper addresses the question of the presence of a major difference in recognition performance with discrete production probabilities compared to continuous ones. Therefore the system is tested using a larger subset of the FERET database. We show that we are able to achieve higher recognition scores and an improvement concerning the computation speed by using discrete modeling techniques.

A spectrum-sharing communication system where the secondary user is aware of the instantaneous channel state information (CSI) of the secondary link, but knows only the statistics of the secondary transmitter-primary receiver link, is investigated. The optimum power profile and the ergodic capacity of the secondary link are derived for general fading channels (with continuous probability density function) under average and peak transmit-power constraints and with respect to two different interference constraints: an interference outage constraint and a signal-to-interference (SI) outage constraint. When applied to Rayleigh fading channels, our results show that the interference constraint is harmful at high-power regime, whereas at low-power regime, it has a marginal impact and no-interference performance may be achieved.

In this work, the design and evaluation of the recognition performance of hidden Markov models (HMM) for the intervocalic Spanish stops are described. The proposed HMMs were context dependent in order to account for coarticulatory effects. Continuous probability density functions were used for the output vectors, which included spectral dynamic parameters. Initial model parameter estimates were obtained by means of an ad hoc segmentation procedure for careful modeling of spectral transitions. The speech database consisted of a total of 2592 productions of the Spanish stops/p, t, k, b, d, g/ in intervocalic position with the vowels /a, i, u/ in all combinations. The speech data were produced by two male Argentine speakers. As a step to evaluate the recognition performance with running speech, three different consonants were produced embedded in VCVCVCV nonsense utterances. Results were obtained from variations of the basic models that consisted of a comparative study of the recognition performance under different degrees of context dependence, and the alternative use of spectral transition parameters. In the best condition the average consonant recognition performance, obtained in a speaker‐dependent manner, was 96.61% and 96.53% for each speaker. The errors were concentrated mainly in the voiced stops, with almost perfect recognition obtained for the unvoiced stops.

We present a novel variational framework for deformable multi-modal image registration. Our approach is based on Renyi's statistical dependence measure of two random variables with the use of reproducing kernel Hilbert spaces associated with Gaussian kernels to simplify the computation. The popularly used method of maximizing mutual information based optimization algorithms are complex and sensitive to the quantization of the intensities, because it requires the estimation of continuous joint probability density function (pdf). The proposed model does not deal with joint pdf but instead observed independent samples. Experimental results are provided to show the effectiveness of the model.

Theoretical approaches to magnitude‐frequency analysis (MFA) of sediment transport in channels couple continuous flow probability density functions (PDFs) with power law flow‐sediment transport relations (rating curves) to produce closed‐form equations relating MFA metrics such as the effective discharge, Qeff, and fraction of sediment transported by discharges greater than Qeff, f+, to statistical moments of the flow PDF and rating curve parameters. These approaches have proven useful in understanding the theoretical drivers behind the magnitude and frequency of sediment transport. However, some of their basic assumptions and findings may not apply to natural rivers and streams with more complex flow‐sediment transport relationships or management and design scenarios, which have finite time horizons. We use simple numerical experiments to test the validity of theoretical MFA approaches in predicting the magnitude and frequency of sediment transport. Median values of Qeff and f+ generated from repeated, synthetic, finite flow series diverge from those produced with theoretical approaches using the same underlying flow PDF. The closed‐form relation for f+ is a monotonically increasing function of flow variance. However, using finite flow series, we find that f+ increases with flow variance to a threshold that increases with flow record length. By introducing a sediment entrainment threshold, we present a physical mechanism for the observed diverging relationship between Qeff and flow variance in fine and coarse‐bed channels. Our work shows that through complex and threshold‐driven relationships sediment transport mode, channel morphology, flow variance, and flow record length all interact to influence estimates of what flow frequencies are most responsible for transporting sediment in alluvial channels.

A general framework for deterministic univariate ensemble filtering is presented. The framework fits a continuous prior probability density function (PDF) to the prior ensemble. A functional representation for the observation likelihood is combined with the prior PDF to get a continuous analysis (posterior) PDF. Cumulative distribution functions for the prior and analysis are also required. The key innovation is that an analysis ensemble is computed so that the quantile of each ensemble member is the same as its prior quantile. Many choices for the prior PDF family and the likelihood function are described. A choice of normal prior with normal likelihood is equivalent to the ensemble adjustment Kalman filter. Some other choices for the prior include gamma, inverse gamma, beta, beta prime, lognormal, and exponential distributions. Both prior distributions and likelihoods can be defined over a set of intervals giving additional flexibility that can be used to implement methods like a Huber likelihood for observations with occasional outliers. Priors and likelihoods can also be defined as sums of distributions allowing choices like bivariate normals or kernel filters. Empirical distributions, for instance piecewise linear approximations to arbitrary PDFs and functions can be used. Another empirical choice leads to the rank histogram filter. Results here are univariate and can be used to compute increments for observed variables or marginal distributions for any variable for a reanalysis. Linear regression of increments can be used to update state variables in a serial filter to build a comprehensive data assimilation system. Part 2 will discuss other methods for extending the framework to multivariate data assimilation. Significance Statement Data assimilation is used to combine information from model forecasts with subsequent observations to obtain better estimates of the current state of the atmosphere or other parts of the Earth system. Ensemble data assimilation uses a number of forecasts to get more information about uncertainty. A new method allows much more flexibility in the assumptions that must be made when doing ensemble data assimilation. As an example, the method can be better for quantities that are bounded like the amount of an atmospheric trace pollutant.