Separating models of learning with faulty teachers

A general technique is developed to obtain nonlearnability results in the model of exact learning from equivalence and membership queries. The technique is applied to show that, assuming NP not=co-NP, there does not exist a polynomial-time membership and equivalence query algorithm for exactly learning read-thrice DNF formulas-boolean formulas in disjunctive normal form where each variable appears at most three times. This result adds evidence to the conjecture that DNF is hard to learn in the membership and equivalence query model.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Exact Learning Boolean Functions via the Monotone Theory

We experimentally demonstrated for the first time in 2019 LiCoO2 all solid-state thin-film batteries with high electrochemical density storage of 0.89 mA.h.cm-2 [1]. Despite these excellent performances, some devices may be defective and fail during cycling. Therefore, in the context of industrial production of such devices, there is a need to reliably assess the functionality of the cells on the wafer scale. The most straightforward approach consists in simply cycling the batteries as well as performing complementary short tests such as Open-Circuit Voltage measurement (OCV) and Electrochemical Impedance Spectroscopy (EIS). However, with 14 000+ cells per wafer said approach is too time consuming. Moreover tested batteries would not be suitable for further commercialization as the electrodes have already been solicited, which could affect their shelf life.We therefore have investigated on the ability of a short – i.e. with a duration in the order of the minute – pulsed test to predict the failure of our dies, using machine learning. This Electrical Wafer Sorting (EWS) or screening protocol consists in inducing a partial charge for 1 min and then to observe the relaxation by measuring the OCV for 10 s. OCV and EIS are also measured before and after the pulse. Certain features are extracted from profiles collected during the EWS protocol (shown in Figure a)), such as the mean OCV or the real part of the impedance at 1 kHz. 1122 devices were screened using this protocol and then cycled for 1 cycle (1 full charge 1 full discharge). The batteries were then classified in binary categories, according to the outcome of the cycling test: fail or pass. Fail category encompasses all failure modes: (e.g. open-circuit, short-circuit). The features extracted during the EWS were used to build the prediction model, whose target is the cycling categorization. Foremost, the data was split in a training set and a test set. In a first step, variable selection was achieved using a simple decision tree model. It enabled us to choose the combination of parameters leading to a maximization of the f1-scores (from 68 to 73 %). However, the learning curves revealed that this simple model clearly overfitted and was unable to generalize. In a second step, a variety of models were trained using the selected features, including Random Forest, Support Vector Machines, and Logistic Regression. The three models leading to best f1-scores were kept and their hyperparameters have been further optimized following a grid search approach. The algorithm giving the highest scores is the Random Forest classifier. Although it was not possible to increase its accuracy (still at 73%) the optimization step helped to significantly reduce the overfitting, reinforcing the ability of the model to be equally accurate on new data. The corresponding confusion matrix and learning curves are displayed on Figure b). Figure c) shows respectively predicted and actual wafermaps for data present in the dataset used to build the model. An example of predicted wafermap on a complete wafer is presented on Figure d).Achieved performance provide an interesting base for investigations in a research context. It is shown that the functionality of the thin film batteries can be predicted with reasonable confidence. Nevertheless, far better scores (above 90%) are required for production application. Thus further work will focus on collecting new data to check on generalizability and if it helps to improve the model accuracy when used for training. All the work presented in this abstract has been carried out using scikit-learn Python package [2].[1] S. Oukassi et al., “Millimeter scale thin film batteries for integrated high energy density storage,” 2019, vol. 2019-December. doi: 10.1109/IEDM19573.2019.8993483.[2] F. Pedregosa et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825–2830, 2011. Figure 1

Hardness of approximate two-level logic minimization and PAC learning with membership queries

The PAC and other equivalent learning models are widely accepted models for polynomial learnability of concept classes. However, negative results abound in the PAC learning framework (concept classes such as deterministic finite state automata (DFA) are not efficiently learnable in the PAC model). The PAC model’s requirement of learnability under all conceivable distributions could be considered too stringent a restriction for practical applications. Several models for learning in more helpful environments have been proposed in the literature including: learning from example based queries [2], online learning allowing a bounded number of mistakes [14], learning with the help of teaching sets [7], learning from characteristic sets [5], and learning from simple examples [12,4]. Several concept classes that are not learnable in the standard PAC model have been shown to be learnable in these models. In this paper we identify the relationships between these different learning models. We also address the issue of unnatural collusion between the teacher and the learner that can potentially trivialize the task of learning in helpful environments.

Learning from positive examples occurs very frequently in natural learning. The PAC learning model of Valiant takes many features of natural learning into account,but in most cases it fails to describe such kind of learning. We show that in order to make the learning from positive data possible, extra-information about the underlying distribution must be provided to the learner. We define a PAC learning model from positive and unlabeled examples. We also define a PAC learning model from positive and unlabeled statistical queries.Relations with PAC model ([Val84]), statistical query model ([Kea93]) and constant-partition classification noise model ([Dec97]) are studied. We show that k DNF and k decision lists are learnable in both models, i.e. with far less information than it is assumed in previously used algorithms.KeywordsLearning AlgorithmBoolean FunctionConcept ClassUnlabeled DataUnderlying DistributionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We introduce a new fault-tolerant model of algorithmic learning using an equivalence oracle and an incomplete membership oracle , in which the answers to a random subset of the learner's membership queries may be missing. We demonstrate that, with high probability, it is still possible to learn monotone DNF formulas in polynomial time, provided that the fraction of missing answers is bounded by some constant. Even when half the membership queries are expected to yield no additional information, our algorithm will exactly identify m -term, n -variable monotone DNF formulas with an expected O ( mn 2 ) queries. The same task has been shown to require exponential time using equivalence queries alone. Thus, this model may lead to a better understanding of the power of membership queries.

We consider the following question: using a source of labeled data and interaction with an untrusted prover, what is the complexity of verifying that a given hypothesis is "approximately correct"? We study interactive proof systems for PAC verification, where a verifier that interacts with a prover is required to accept good hypotheses, and reject bad hypotheses. Both the verifier and the prover are efficient and have access to labeled data samples from an unknown distribution. We are interested in cases where the verifier can use significantly less data than is required for (agnostic) PAC learning, or use a substantially cheaper data source (e.g., using only random samples for verification, even though learning requires membership queries). We believe that today, when data and data-driven algorithms are quickly gaining prominence, the question of verifying purported outcomes of data analyses is very well-motivated. We show three main results. First, we prove that for a specific hypothesis class, verification is significantly cheaper than learning in terms of sample complexity, even if the verifier engages with the prover only in a single-round (NP-like) protocol. Moreover, for this class we prove that single-round verification is also significantly cheaper than testing closeness to the class. Second, for the broad class of Fourier-sparse boolean functions, we show a multi-round (IP-like) verification protocol, where the prover uses membership queries, and the verifier is able to assess the result while only using random samples. Third, we show that verification is not always more efficient. Namely, we show a class of functions where verification requires as many samples as learning does, up to a logarithmic factor.

In the PAC learning model, the learner must output good approximate results with only the information that tells the learner whether the example is “positive” or “negative”. This restriction results in narrowing the area of the learnable concepts or in increasing the size of examples required for successful learning. However, if the learner receives some additional information about the example besides being positive or negative, e.g., real values corresponding to the degree of the positiveness (or negativeness), a larger class may become learnable or the number of necessary examples may be reduced.In the case of learning geometric concepts, such additional information may be given, for instance, how close a given positive or negative example is to the boundary of a target geometric concept. Using this type of additional information for geometric concepts consisting of complex boundaries, the learner may identify the geometric concept more rigorously or with sampling the smaller number of examples for the required accuracy. In the case of neural networks of threshold functions, some of values of weighted sum of inputs at nodes, say a value at the output node, may be output instead of simply producing outputs of either 0 or 1 after comparing the values with the threshold values at nodes. Such values represent the closeness of an input to the threshold boundary.This note investigates the effect of such additional information in computational learning theory. Two types of networks of threshold-like functions are considered, and it is demonstrated how some additional information is profitable to learn these networks more efficiently.KeywordsHide LayerHide UnitThreshold FunctionConvex PolytopesVoronoi RegionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A PAC model under helpful distributions is introduced. A teacher associates a teaching set with each target concept and we only consider distributions such that each example in the teaching set has a non-zero weight. The performance of a learning algorithm depends on the probabilities of the examples in this teaching set. In this model, an Occam's razor theorem and its converse are proved. The class of decision lists is proved PAC learnable under helpful distributions. A PAC learning model with simple teacher (simplicity is based on program-size complexity) is also defined and the model is compared with other models of teaching.

In the last few decades a computational approach to machine learning has emerged based on paradigms from recursion theory and the theory of computation. Such ideas include learning in the limit, learning by enumeration, and probably approximately correct (pac) learning. These models usually are not suitable in practical situations. In contrast, statistics based inference methods have enjoyed a long and distinguished career. Currently, Bayesian reasoning in various forms, minimum message length (MML) and minimum description length (MDL), are widely applied approaches. They are the tools to use with particular machine learning praxis such as simulated annealing, genetic algorithms, genetic programming, artificial neural networks, and the like. These statistical inference methods select the hypothesis which minimizes the sum of the length of the description of the hypothesis (also called ‘model’) and the length of the description of the data relative to the hypothesis. It appears to us that the future of computational machine learning will include combinations of the approaches above coupled with guaranties with respect to used time and memory resources. Computational learning theory will move closer to practice and the application of the principles such as MDL require further justification. Here, we survey some of the actors in this dichotomy between theory and praxis, we justify MDL via the Bayesian approach, and give a comparison between pac learning and MDL learning of decision trees.

We study the learning models defined by Angluin et al. (1997): learning with equivalence and limited membership queries and learning with equivalence and malicious membership queries. We show that if a class of concepts that is closed under projection is learnable in polynomial time using equivalence and (standard) membership queries then it is learnable in polynomial time in the above models. This closes the open problems by Angluin et al. (1997). Our algorithm can also handle errors in the equivalence queries.

Supervised learning—that is, learning from labeled examples—is an area of Machine Learning that has reached substantial maturity. It has generated general-purpose and practically successful algorithms and the foundations are quite well understood and captured by theoretical frameworks such as the PAC-learning model and the Statistical Learning theory framework. However, for many contemporary practical problems such as classifying web pages or detecting spam, there is often additional information available in the form of unlabeled data, which is often much cheaper and more plentiful than labeled data. As a consequence, there has recently been substantial interest in semi-supervised learning—using unlabeled data together with labeled data—since any useful information that reduces the amount of labeled data needed can be a significant benefit. Several techniques have been developed for doing this, along with experimental results on a variety of different learning problems. Unfortunately, the standard learning frameworks for reasoning about supervised learning do not capture the key aspects and the assumptions underlying these semi -supervised learning methods. In this article, we describe an augmented version of the PAC model designed for semi-supervised learning, that can be used to reason about many of the different approaches taken over the past decade in the Machine Learning community. This model provides a unified framework for analyzing when and why unlabeled data can help, in which one can analyze both sample-complexity and algorithmic issues. The model can be viewed as an extension of the standard PAC model where, in addition to a concept class C , one also proposes a compatibility notion: a type of compatibility that one believes the target concept should have with the underlying distribution of data. Unlabeled data is then potentially helpful in this setting because it allows one to estimate compatibility over the space of hypotheses, and to reduce the size of the search space from the whole set of hypotheses C down to those that, according to one's assumptions, are a-priori reasonable with respect to the distribution. As we show, many of the assumptions underlying existing semi-supervised learning algorithms can be formulated in this framework. After proposing the model, we then analyze sample-complexity issues in this setting: that is, how much of each type of data one should expect to need in order to learn well, and what the key quantities are that these numbers depend on. We also consider the algorithmic question of how to efficiently optimize for natural classes and compatibility notions, and provide several algorithmic results including an improved bound for Co-Training with linear separators when the distribution satisfies independence given the label.

A revision algorithm is a learning algorithm that identifies the target concept, starting from an initial concept. Such an algorithm is considered efficient if its complexity (in terms of the measured resource) is polynomial in the syntactic distance between the initial and the target concept, but only polylogarithmic in the number of variables in the universe. We give efficient revision algorithms in the model of learning with equivalence and membership queries. The algorithms work in a general revision model where both deletion and addition revision operators are allowed. In this model one of the main open problems is the efficient revision of Horn formulas. Two revision algorithms are presented for special cases of this problem: for depth-1 acyclic Horn formulas, and for definite Horn formulas with unique heads.

We extend the line of research initiated by Fortnow and Klivans [6] that studies the relationship between efficient learning algorithms and circuit lower bounds. In [6], it was shown that if a Boolean circuit class \(\mathcal{C}\) has an efficient deterministic exact learning algorithm, (i.e. an algorithm that uses membership and equivalence queries) then \(\mathsf{EXP}^{\mathsf{NP}} \not \subseteq \mathsf{P/poly}[\mathcal{C}]\). Recently, in [14] EXP NP was replaced by DTIME(n ω(1)). Yet for the models of randomized exact learning or Valiant’s PAC learning, the best result so far is a lower bound against BPEXP (the exponential-time analogue of BPP). In this paper, we derive stronger lower bounds as well as some other consequences from randomized exact learning and PAC learning algorithms, answering an open question posed in [6] and [14]. In particular, we show that