Factorizable Joint Shift in Multinomial Classification

Labeled data for classification could often be obtained by sampling that restricts or favors choice of certain classes. A classifier trained using such data will be biased, resulting in wrong inference and sub-optimal classification on new data. Given an unlabeled new data set we propose a bootstrap method to estimate its class probabilities by using an estimate of the classifier’s accuracy on training data and an estimate of probabilities of classifier’s predictions on new data. Then, we propose two methods to improve classification accuracy on new data. The first method can be applied only if a classifier was designed to predict posterior class probabilities where predictions of an existing classifier are adjusted according to the estimated class probabilities of new data. The second method can be applied to an arbitrary classification algorithm, but it requires retraining on the properly resampled data. The proposed bootstrap algorithm was validated through experiments with 500 replicates calculated on 1,000 realizations for each of 16 choices of data set size, number of classes, prior class probabilities and conditional probabilities describing a classifier’s performance. Applications of the proposed methodology to a benchmark data set with various class probabilities on unlabeled data and balanced class probabilities on the training data provided strong evidence that the proposed methodology can be successfully used to significantly improve classification on unlabeled data.

Researchers in the fields of neural networks, statistics, machine learning, and artificial intelligence have followed three basic approaches to developing new pattern classifiers. Probability Density Function (PDF) classifiers include Gaussian and Gaussian Mixture classifiers which estimate distributions or densities of input features separately for each class. Posterior probability classifiers include multilayer perceptron neural networks with sigmoid nonlinearities and radial basis function networks. These classifiers estimate minimum-error Bayesian a posteriori probabilities (hereafter referred to as posterior probabilities) simultaneously for all classes. Boundary forming classifiers include hard-limiting single-layer perceptrons, hypersphere classifiers, and nearest neighbor classifiers. These classifiers have binary indicator outputs which form decision regions that specify the class of any input pattern. Posterior probability and boundary-forming classifiers are trained using discriminant training. All training data is used simultaneously to estimate Bayesian posterior probabilities or minimize overall classification error rates. PDF classifiers are trained using maximum likelihood approaches which individually model class distributions without regard to overall classification performance. Analytic results are presented which demonstrate that many neural network classifiers can accurately estimate posterior probabilities and that these neural network classifiers can sometimes provide lower error rates than PDF classifiers using the same number of trainable parameters. Experiments also demonstrate how interpretation of network outputs as posterior probabilities makes it possible to estimate the confidence of a classification decision, compensate for differences in class prior probabilities between test and training data, and combine outputs of multiple classifiers over time for speech recognition.

Despite the fact that real-life data streams may often be characterized by the dynamic changes in the prior class probabilities, there is a scarcity of articles trying to clearly describe and classify this problem as well as suggest new methods dedicated to resolving this issue. The following paper aims to fill this gap by proposing a novel data stream taxonomy defined in the context of prior class probability and by introducing the Dynamic Statistical Concept Analysis (DSCA) - prior probability estimation algorithm. The proposed method was evaluated using computer experiments carried out on 100 synthetically generated data streams with various class imbalance characteristics. The obtained results, supported by statistical analysis, confirmed the usefulness of the proposed solution, especially in the case of discrete dynamically imbalanced data streams (DDIS).

During the last years, speech recognition technologies have started their migration from research laboratories to real word applications gaining market shares. Although this shows that paradigms like Neural Networks have reached a high level of accuracy in modeling speech, it must be realized that there is still room for improving recognition performances exploiting the feedbacks coming from the applicative fields. In these cases, in fact, precious application dependent speech material can be recorded, and used to train the acoustic models in order to improve the behaviour of the recognizer on target dictionaries. The best results can be achieved when an iterative, refining process is set up. Unfortunately, speech corpora coming from the field are seldom phonetically balanced and this can cause the performances of the Neural Network to get worse, wasting the benefits of the refining process. In this paper, the problem of Prior Probability normalization has been faced and a method for Prior Probability normalization has been investigated, with the important characteristic of being applicable simply through a modification of the biases at the end of the training phase (therefore on trained nets). An experimentation on several languages is reported, showing the Prior Probability normalization seems quite useful to improve recognition accuracy and to get rid of some undesired effects of training data-bases not perfectly phonetically balanced.

Objective: To compare the performance of multiple machine learning algorithms in predicting recurrence after resection of early-stage hepatocellular carcinoma(HCC). Methods: Clinical data of 882 early-stage HCC patients who were admitted to the First Affiliated Hospital of Nanjing Medical University from May 2009 to December 2019 and treated with curative surgical resection were retrospectively collected. There were 701 males and 181 females,with an age of (57.3±10.5)years(range:21 to 86 years). All patients were randomly assigned in a 2∶1 ratio, the training dataset consisted of 588 patients and the test dataset consisted of 294 patients. The construction of machine learning-based prediction models included random survival forest(RSF),gradient boosting machine,elastic net regression and Cox regression model. The prediction accuracy of the model was measured by the concordance index(C-index). The prediction error of the model was measured by the integrated Brier score. Model fit was assessed by the calibration plot. The performance of machine learning models with that of rival model and HCC staging systems was compared. All models were validated in the independent test dataset. Results: Median recurrence-free survival was 61.7 months in the training dataset while median recurrence-free survival was 61.9 months in the validation dataset, there was no significant difference between two datasets in terms of recurrence-free survival(χ²=0.029,P=0.865). The RSF model consisted of 5 commonly used clinicopathological characteristics, including albumin-bilirubin grade,serum alpha fetoprotein,tumor number,type of hepatectomy and microvascular invasion. In both training and test datasets,the RSF model provided the best prediction accuracy,with respective C-index of 0.758(95%CI:0.725 to 0.791) and 0.749(95%CI:0.700 to 0.797),and the lowest prediction error,with respective integrated Brier score of 0.171 and 0.151. The prediction accuracy of RSF model for recurrence after resection of early-stage HCC was superior to that of other machine learning models,rival model(ERASL model) as well as HCC staging systems(BCLC,CNLC and TNM staging),with statistically significant difference(P<0.01). Calibration curves demonstrated good agreement between RSF model-predicted probabilities and observed outcomes.All patients could be stratified into low-risk,intermediate-risk or high-risk group based on RSF model;statistically significant differences among three risk groups were observed in both training and test datasets(all P<0.01). The risk stratification of RSF model was superior to that of TNM staging. Conclusion: The proposed RSF model assembled with 5 commonly used clinicopathological characteristics in this study can predict the recurrence risk with favorable accuracy that may facilitate clinical decision-support for patients with early-stage HCC.

A simple dynamic model of a neural network is presented. Using the dynamic model of a neural network, we improve the performance of a three-layer multilayer perceptron (MLP). The dynamic model of a MLP is used to make fundamental changes in the network optimization strategy. These changes are: Neuron activation functions are used, which reduce the probability of singular Jacobians; Successive regularization is used to constrain the volume of the weight space being minimized; Boltzmann pruning is used to constrain the dimension of the weight space; and prior class probabilities are used to normalize all error calculations, so that statistically significant samples of rare but important classes can be included without distortion of the error surface. All four of these changes are made in the inner loop of a conjugate gradient optimization iteration and are intended to simplify the training dynamics ofthe optimization. On handprinted digits and fingerprint classification problems, these modifications improve error-reject performance by factors between 2 and 4 and reduce network size by 40 to 60%.

The article introduces how to validate regression models in the analysis of competing risks. The prediction accuracy of competing risks regression models can be assessed by discrimination and calibration. The area under receiver operating characteristic curve (AUC) or Concordance-index, and calibration plots have been widely used as measures of discrimination and calibration, respectively. One-time splitting method can be used for randomly splitting original data into training and test datasets. However, this method reduces sample sizes of both training and testing datasets, and the results can be different by different splitting processes. Thus, the cross-validation method is more appealing. For time-to-event data, model validation is performed at each analysis time point. In this article, we review how to perform model validation using the riskRegression package in R, along with plotting a nomogram for competing risks regression models using the regplot() package.

Real-time probabilistic classification of fire and smoke using thermal imagery for intelligent firefighting robot

Ovarian cancer (OC) is a frequent and fatal disease in women, and bone metastasis of ovarian cancer (OCBM) leads to a poor survival trend. This study aimed to determine the factors which influence overall survival (OS) and cancer-specific survival (CSS) of OCBM patients and to develop prognostic predictive models. Data of OCBM patients were stratified from the Surveillance, Epidemiology and End Results database from 2010 to 2017 and were randomly divided into training and testing datasets (7:3). Prognostic factors were identified by Cox regression analyses and nomograms were then developed. Nomogram models were examined on the discriminative ability and accuracy by calibration plots, Brier score (BS), and time-dependent receiver operating characteristic (ROC) curves. Decision curve analyses (DCA) was used for estimation of the clinical benefit of nomogram models. Grade, tumor size, tumor metastasis (liver, lung), primary site surgery, chemotherapy, and systemic therapy were realized as independent prognostic factors for OS and CSS, respectively. Agreement between the actual and predicted outcomes was proved by calibration plots. Nomograms performed well in OS and CSS predictions, as shown by area under the ROC curves (AUCs) and BSs for testing dataset as follows: for OS, 3-/6-/12-month AUCs and BSs were 0.778/0.788/0.822 and 19.0/18.5/15.4, respectively; for CSS, 3-/6-/12-month AUCs and BSs were 0.799/0.806/0.832 and 18.1/18.0/15.4, respectively. DCA suggested an agreeable clinical benefit of both nomograms. The nomograms developed for OCBM patients' survival prediction were proved to be accurate, efficient, and clinically beneficial, which were further deployed as web-based calculators to help in clinical decision making and future studies.

At the start of new operation, a production process is commonly unstable. Then, a process condition is into being stable gradually over time. For the purpose of shifting to mass production, the stability in a process should be evaluated. This paper proposes a method of evaluating the process stability based on Bayesian statistics and information theory. In Bayesian statistics, the knowledge for a process is renewed by deriving posterior distribution based on current prior distribution and observations. Here, equivalency between prior and posterior distributions could be considered to be a criterion for the stability in a process. It is needed to define difference between prior and posterior distributions in order to evaluate the equivalency between prior and posterior distributions. We first formulate the relation between the prior distribution and the posterior distribution for process mean using Bayesian statistics. Secondly, we evaluate the difference between the both distributions based on Kullback‐Leibler (K‐L) divergence in information theory. Finally, some numerical examples in the method of evaluating the stability in process mean are illustrated. Also, we discuss about a decision rule for the stability in process mean through the numerical investigation.

A fuzzy-logic approach to the classification of multitemporal, multisensor remote-sensing images is proposed. The approach is based on a fuzzy fusion of three basic sources of information: spectral, spatial and temporal contextual information sources. It aims at improving the accuracy over that of single-time noncontextual classification. Single-time class posterior probabilities, which are used to represent spectral information, are estimated by Multilayer Perceptron neural networks trained for each single-time image, thus making the approach applicable to multisensor data. Both the spatial and temporal kinds of contextual information are derived from the single-time classification maps obtained by the neural networks. The expert's knowledge of possible transitions between classes at two different times is exploited to extract temporal contextual information. The three kinds of information are then fuzzified in order to apply a fuzzy reasoning rule for their fusion. Fuzzy reasoning is based on the "MAX" fuzzy operator and on information about class prior probabilities. Finally, the class with the largest fuzzy output value is selected for each pixel in order to provide the final classification map. Experimental results on a multitemporal data set consisting of two multisensor (Landsat TM and ERS-1 SAR) images are reported. The accuracy of the proposed fuzzy spatio-temporal contextual classifier is compared with those obtained by the Multilayer Perceptron neural networks and a reference classification approach based on Markov Random Fields (MRFs). Results show the benefit of adding spatio-temporal contextual information to the classification scheme, and suggest that the proposed approach represents an interesting alternative to the MRF-based approach, in particular, in terms of simplicity.

This paper presents a novel active learning method to detect land-cover transitions, which is defined in the framework of the Bayes rule for compound classification. Compound classification is a supervised technique that requires a suitable multitemporal training set for modeling the temporal correlation between multitemporal images. The temporal correlation is represented by the prior joint probabilities of classes which allow one to obtain accurate land-cover transitions maps. However, the collection of labeled samples is time consuming as well as costly. In this paper, a novel active learning method based on joint entropy is proposed to properly increase the number of initial multitemporal training samples by taking into account the temporal correlation between multitemporal images. Experimental results confirmed the effectiveness of the proposed joint entropy based active learning method for compound classification.

Automatic lithology prediction from well logging using kernel density estimation

&lt;p&gt;&lt;strong&gt;Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.&lt;/p&gt;&lt;p&gt;We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.&lt;/p&gt;&lt;p&gt;We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.&lt;/p&gt;&lt;p&gt;We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β &lt; 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β &gt; 1 to β &lt; 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.&lt;/p&gt;&lt;p&gt;In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.&lt;/p&gt;

Deep Neural Networks (DNNs) often demonstrate remarkable performance when evaluated on the test dataset used during model creation. However, their ability to generalize effectively when deployed is crucial, especially in critical applications. One approach to assess the generalization capability of a DNN model is to evaluate its performance on replicated test datasets, which are created by closely following the same methodology and procedures used to generate the original test dataset. Our investigation focuses on the performance degradation of pre-trained DNN models in multi-class classification tasks when evaluated on these replicated datasets; this performance degradation has not been entirely explained by generalization shortcomings or dataset disparities. To address this, we introduce a new evaluation framework that leverages uncertainty estimates generated by the models studied. This framework is designed to isolate the impact of variations in the evaluated test datasets and assess DNNs based on the consistency of their confidence in their predictions. By employing this framework, we can determine whether an observed performance drop is primarily caused by model inadequacy or other factors. We applied our framework to analyze 564 pre-trained DNN models across the CIFAR-10 and ImageNet benchmarks, along with their replicated versions. Contrary to common assumptions about model inadequacy, our results indicate a substantial reduction in the performance gap between the original and replicated datasets when accounting for model uncertainty. This suggests a previously unrecognized adaptability of models to minor dataset variations. Our findings emphasize the importance of understanding dataset intricacies and adopting more nuanced evaluation methods when assessing DNN model performance. This research contributes to the development of more robust and reliable DNN models, especially in critical applications where generalization performance is of utmost importance. The code to reproduce our experiments will be available at https://github.com/esla/Reassessing_DNN_Accuracy.