On Using Extended Statistical Queries to Avoid Membership Queries

A recent innovation in computational learning theory is the statistical query (SQ) model. The advantage of specifying learning algorithms in this model is that SQ algorithms can be simulated in the probably approximately correct (PAC) model, both in the absenceandin the presence of noise. However, simulations of SQ algorithms in the PAC model have non-optimal time and sample complexities. In this paper, we introduce a new method for specifying statistical query algorithms based on a type ofrelative errorand provide simulations in the noise-free and noise-tolerant PAC models which yield more efficient algorithms. Requests for estimates of statistics in this new model take the following form: “Return an estimate of the statistic within a 1±?factor, or return ?, promising that the statistic is less than?.” In addition to showing that this is a very natural language for specifying learning algorithms, we also show that this new specification is polynomially equivalent to standard SQ, and thus, known learnability and hardness results for statistical query learning are preserved. We then give highly efficient PAC simulations of relative error SQ algorithms. We show that the learning algorithms obtained by simulating efficient relative error SQ algorithms both in the absence of noise and in the presence of malicious noise have roughly optimal sample complexity. We also show that the simulation of efficient relative error SQ algorithms in the presence of classification noise yields learning algorithms at least as efficient as those obtained through standard methods, and in some cases improved, roughly optimal results are achieved. The sample complexities for all of these simulations are based on thed?metric, which is a type of relative error metric useful for quantities which are small or even zero. We show that uniform convergence with respect to thed?metric yields “uniform convergence” with respect to (?, ?) accuracy. Finally, while we show that manyspecificlearning algorithms can be written as highly efficient relative error SQ algorithms, we also show, in fact, thatallSQ algorithms can be written efficiently by proving general upper bounds on the complexity of (?, ?) queries as a function of the accuracy parameter?. As a consequence of this result, we give general upper bounds on the complexity of learning algorithms achieved through the use of relative error SQ algorithms and the simulations described above.

Hardness of approximate two-level logic minimization and PAC learning with membership queries

We derive general bounds on the complexity of learning in the statistical query model and in the PAC model with classification noise. We do so by considering the problem of boosting the accuracy of weak learning algorithms which fall within the statistical query model. This new model was introduced by M. Kearns (1993) to provide a general framework for efficient PAC learning in the presence of classification noise. >

Learning from positive examples occurs very frequently in natural learning. The PAC learning model of Valiant takes many features of natural learning into account,but in most cases it fails to describe such kind of learning. We show that in order to make the learning from positive data possible, extra-information about the underlying distribution must be provided to the learner. We define a PAC learning model from positive and unlabeled examples. We also define a PAC learning model from positive and unlabeled statistical queries.Relations with PAC model ([Val84]), statistical query model ([Kea93]) and constant-partition classification noise model ([Dec97]) are studied. We show that k DNF and k decision lists are learnable in both models, i.e. with far less information than it is assumed in previously used algorithms.KeywordsLearning AlgorithmBoolean FunctionConcept ClassUnlabeled DataUnderlying DistributionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper we develop a new approach for learning decision trees and multivariate polynomials via interpolation of multivariate polynomials. This new approach yields simple learning algorithms for multivariate polynomials and decision trees over finite fields under any constant bounded product distribution. The output hypothesis is a (single) multivariate polynomial that is an /spl epsiv/-approximation of the target under any constant bounded product distribution. The new approach demonstrates the learnability of many classes under any constant bounded product distribution and using membership queries, such as j-disjoint DNF and multivariate polynomial with bounded degree over any field. The technique shows how to interpolate multivariate polynomials with bounded term size from membership queries only. This in particular gives a learning algorithm for O(log n)-depth decision tree from membership queries only and a new learning algorithm of any multivariate polynomial over sufficiently large fields from membership queries only. We show that our results for learning from membership queries only are the best possible.

Article Free Access Share on Asking questions to minimize errors Authors: Nader H. Bshouty View Profile , Sally A. Goldman View Profile , Thomas R. Hancock View Profile , Sleiman Matar View Profile Authors Info & Claims COLT '93: Proceedings of the sixth annual conference on Computational learning theoryAugust 1993 Pages 41–50https://doi.org/10.1145/168304.168310Published:01 August 1993Publication History 17citation179DownloadsMetricsTotal Citations17Total Downloads179Last 12 Months9Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

A new research frontier in AI and data mining seeks to develop methods to automatically discover relevant variables among many irrelevant ones. In this paper, we present four algorithms that output such crucial variables in PAC model with membership queries. The first algorithm executes the task under any unknown distribution by measuring the distance between virtual and real targets. The second algorithm exhausts virtual version space under an arbitrary distribution. The third algorithm exhausts universal set under the uniform distribution. The fourth algorithm measures influence of variables under the uniform distribution. Knowing the number r of relevant variables, the first algorithm runs in almost linear time for r. The second and the third ones use less membership queries than the first one, but runs in time exponential for r. The fourth one enumerates highly influential variables in quadratic time for r.

We present an algorithm to learn a deterministic timed automaton (DTA) via membership and equivalence queries. Our algorithm is an extension of the L* algorithm with a Myhill-Nerode style characterization of recognizable timed languages, which is the class of timed languages recognizable by DTAs. We first characterize the recognizable timed languages with a Nerode-style congruence. Using it, we give an algorithm with a smart teacher answering symbolic membership queries in addition to membership and equivalence queries. With a symbolic membership query, one can ask the membership of a certain set of timed words at one time. We prove that for any recognizable timed language, our learning algorithm returns a DTA recognizing it. We show how to answer a symbolic membership query with finitely many membership queries. We also show that our learning algorithm requires a polynomial number of queries with a smart teacher and an exponential number of queries with a normal teacher. We applied our algorithm to various benchmarks and confirmed its effectiveness with a normal teacher.

Polynomial time learning of simple deterministic languages via queries and a representative sample

In this chapter we investigate practical technologies for security and privacy in data analysis at large scale. We motivate our approach by discussing the challenges and opportunities in light of current and emerging analysis paradigms on large data sets. In particular, we present a framework for privacy-preserving distributed data analysis that is practical for many real-world applications. The framework is called Peers for Privacy (P4P) and features a novel heterogeneous architecture and a number of efficient tools for performing private computation and offering security at large scale. It maintains three key properties, which are essential for real-world applications: (i) provably strong privacy; (ii) adequate efficiency at reasonably large scale; and (iii) robustness against realistic adversaries. The framework gains its practicality by decomposing data mining algorithms into a sequence of vector addition steps, which can be privately evaluated using efficient cryptographic tools, namely verifiable secret sharing over small field (e.g., 32 or 64 bits), which have the same cost as regular, non-private arithmetic. This paradigm supports a large number of statistical learning algorithms, including SVD, PCA, k-means, ID3 and machine learning algorithms based on Expectation-Maximization, as well as all algorithms in the statistical query model (Kearns, Efficient noise-tolerant learning from statistical queries. In: STOC’93, San Diego, pp. 392–401, 1993). As a concrete example, we show how singular value decomposition, which is an extremely useful algorithm and the core of many data mining tasks, can be performed efficiently with privacy in P4P. Using real data, we demonstrate that P4P is orders of magnitude faster than other solutions.

An Efficient Membership-Query Algorithm for Learning DNF with Respect to the Uniform Distribution

We prove two lower bounds in the statistical query (SQ) learning model. The first lower bound is on weak-learning. We prove that for a concept class of SQ-dimension d, a running time of @W(d/logd) is needed. The SQ-dimension of a concept class is defined to be the maximum number of concepts that are ''uniformly correlated'', in that each of their pair has nearly the same correlation. This lower bound matches the upper bound in Blum et al. (Weakly Learning DNF and Characterizing Statistical Query Learning using Fourier Analysis, STOC 1994, pp. 253-262), up to a logarithmic factor. We prove this lower bound against an ''honest SQ-oracle'', which gives a stronger result than the ones against the more frequently used ''adversarial SQ-oracles''. The second lower bound is more general. It gives a continuous trade-off between the ''advantage'' of an algorithm in learning the target function and the number of queries it needs to make, where the advantage of an algorithm is the probability it succeeds in predicting a label minus the probability it does not. Both lower bounds extend and/or strengthen previous results, and solve an open problem left in previous papers. An earlier version of this paper [K. Yang, New lower bounds for statistical query learning, in: The Proceedings of the 15th Annual Conference on Computational Learning Theory, COLT 2002, Sydney, Australia, July 8-10, Lecture Notes in Computer Science, vol. 2375, 2002, pp. 229-243.] appeared in the proceedings of the 15th Annual Conference on Computational Learning Theory (COLT 2002).

Exact Learning Boolean Functions via the Monotone Theory

Diabetes mellitus is characterized by excessive blood glucose that occurs when the pancreas malfunctions while producing insulin. High blood glucose levels can cause chronic damage to organs, particularly the eyes and kidneys. Diabetes prediction models traditionally use a variety of machine learning (ML) algorithms by combining data from the glucose levels, patient health parameters, and other biomarkers. Prior research on diabetes prediction using various algorithms, such as support vector machine (SVM) and decision tree (DT) models, demonstrates an accuracy rate of approximately 70%, which is relatively modest. Therefore, in this study, a classification and regression tree (CART) multiclassifier model has been proposed to improve the accuracy of diabetes prediction, which is based on three classes: non-diabetic, pre-diabetic, and diabetic. The study involved data preprocessing steps, hyperparameter tuning, and evaluation of performance metrics. The model achieved 97% accuracy while utilizing the value of 5 for the number of leaves per node, the value of 10 for the maximum number of splits, and deviance as the split criterion, which also resulted in a precision of 98%, recall of 97%, and F1-score of 98%, showing that the proposed multiclassifier model can accurately predict diabetes. In conclusion, the proposed CART model with the best hyperparameter setting can enable the highest accuracy in predicting diabetes classes.

Recent advances in machine learning techniques have significantly influenced the development of early diabetes detection systems. In this paper, different machine learning (ML) algorithms are employed for diabetes detection based on hand movement. The spatial and temporal information for horizontal and vertical hand movement deviations for both diabetic and non-diabetic subjects are obtained using a data acquisition system. A dataset is constructed based on the obtained hand movement information along with other related attributes such as age, gender, weight and vision health. Binary classification of diabetic and non-diabetic subjects is achieved using several machine learning algorithms. These algorithms include Naïve Bayes (NB), Logistic Regression (LR), Linear Discriminate Analysis (LDA), Classification and Regression Tree (CART), k-Nearest Neighbors (KNN), Support Vector Machine (SVM) and Multi-layer Perceptron (MLP). The considered ML algorithms are evaluated and ranked according to a set of key performance metrics that include accuracy, precision, recall and F-1 score. Algorithm ranking results demonstrate that LR, CART, KNN and MLP algorithms have achieved higher ranking positions as compared to other algorithms. The best ranked ML algorithms are then validated and thoroughly investigated to prove their ability to discriminate between diabetic and non-diabetic subjects. Validation results show that the best-ranked ML algorithms have achieved high classification accuracy, precision, recall and F-1 scores. Therefore, the considered machine learning algorithms can serve as efficient tools for diabetes detection based on the constructed dataset.