New Lower Bounds for Statistical Query Learning

A recent innovation in computational learning theory is the statistical query (SQ) model. The advantage of specifying learning algorithms in this model is that SQ algorithms can be simulated in the probably approximately correct (PAC) model, both in the absenceandin the presence of noise. However, simulations of SQ algorithms in the PAC model have non-optimal time and sample complexities. In this paper, we introduce a new method for specifying statistical query algorithms based on a type ofrelative errorand provide simulations in the noise-free and noise-tolerant PAC models which yield more efficient algorithms. Requests for estimates of statistics in this new model take the following form: “Return an estimate of the statistic within a 1±?factor, or return ?, promising that the statistic is less than?.” In addition to showing that this is a very natural language for specifying learning algorithms, we also show that this new specification is polynomially equivalent to standard SQ, and thus, known learnability and hardness results for statistical query learning are preserved. We then give highly efficient PAC simulations of relative error SQ algorithms. We show that the learning algorithms obtained by simulating efficient relative error SQ algorithms both in the absence of noise and in the presence of malicious noise have roughly optimal sample complexity. We also show that the simulation of efficient relative error SQ algorithms in the presence of classification noise yields learning algorithms at least as efficient as those obtained through standard methods, and in some cases improved, roughly optimal results are achieved. The sample complexities for all of these simulations are based on thed?metric, which is a type of relative error metric useful for quantities which are small or even zero. We show that uniform convergence with respect to thed?metric yields “uniform convergence” with respect to (?, ?) accuracy. Finally, while we show that manyspecificlearning algorithms can be written as highly efficient relative error SQ algorithms, we also show, in fact, thatallSQ algorithms can be written efficiently by proving general upper bounds on the complexity of (?, ?) queries as a function of the accuracy parameter?. As a consequence of this result, we give general upper bounds on the complexity of learning algorithms achieved through the use of relative error SQ algorithms and the simulations described above.

We derive general bounds on the complexity of learning in the statistical query model and in the PAC model with classification noise. We do so by considering the problem of boosting the accuracy of weak learning algorithms which fall within the statistical query model. This new model was introduced by M. Kearns (1993) to provide a general framework for efficient PAC learning in the presence of classification noise. >

The correspondence between partial metrics and semivaluations

The complexity of shortest path and dilation bounded interval routing

Hybrid commitments and their applications to zero-knowledge proof systems

We show that if Arthur-Merlin protocols can be derandomized, then there is a Boolean function computable in deterministic exponential-time with access to an $\bm{\NP}$ oracle, that cannot be computed by Boolean circuits of {\em exponential} size. More formally, if $\bm{\prAM \subseteq \P^{\NP}}$ then there is a Boolean function in $\bm{\E^{\NP}}$ that requires circuits of size $\bm{2^{\Omega(n)}}$. $\bm{\prAM}$ is the class of promise problems that have Arthur-Merlin protocols, $\bm{\P^{\NP}}$ is the class of functions that can be computed in deterministic polynomial-time with an $\bm{\NP}$ oracle and $\bm{\E^{\NP}}$ is its exponential analogue. The lower bound in the conclusion of our theorem suffices to construct very strong pseudorandom generators. We also show that the same conclusion holds if the problem of approximate counting the number of accepting paths of a nondeterministic Turing machine up to multiplicative factors can be done in nondeterministic polynomial-time. In other words, showing nondeterministic fully polynomial-time approximation schemes for $\bm{\sharp \P}$-complete problems require proving exponential-size circuit lower bounds. A few works have already shown that if we can find efficient deterministic solutions to some specific tasks (or classes) that are known to be solvable efficiently by randomized algorithms (or proofs), then we obtain lower bounds against certain circuit models. These lower bounds were only with respect to polynomial-size circuits {\em even if full derandomization is assumed}. Thus they only implied fairly weak pseudorandom generators (if at all). A key ingredient in our proof is a connection between computational learning theory and exponential-size lower bounds. We show that the existence of deterministic learning algorithms with certain properties implies exponential-size lower bounds, where the complexity of the hard function is related to the complexity of the learning algorithm.

On the complexity of the reflected logic of proofs

Some recent work [7, 14, 15] in computational learning theory has discussed learning in situations where the teacher is helpful, and can choose to present carefully chosen sequences of labelled examples to the learner. We say a function t in a set H of functions (a hypothesis space) defined on a set X is specified by S***X if the only function in H which agrees with t on S is t itself. The specification number s(t) of t is the least cardinality of such an S. For a general hypothesis space, we show that the specification number of any hypotheis is at least equal to a parameter from [14] known as the testing dimension of H. We investigate in some detail the specification numbers of hypotheses in the set Hn of linearly separable boolean functions: We present general methods for finding upper bounds on s(t) and we characterise those t which have largest s(t). We obtain a general lower bound on the number of examples required and we show that for all nested hypotheses, this lower bound is attained. We prove that for any t e Hn, there is exactly one set of examples of minimal cardinality (i.e., of cardinality s(t)) which specifies t. We then discuss those t e Hn which have limited dependence, in the sense that some of the variables are redundant (i.e., there are irrelevant attributes), giving tight upper and lower bounds on s(t) for such hypotheses. In the final section of the paper, we address the complexity of computing specification numbers and related parameters.

Inplace run-length 2d compressed search

Elimination of spatial connectives in static spatial logics

The minimum broadcast range assignment problem on linear multi-hop wireless networks

Deterministic monotone algorithms for scheduling on related machines

A subexponential algorithm for discrete logarithms over hyperelliptic curves of large genus over GF( q)

On the Use of Automata-based Techniques in Symbolic Model Checking: Invited Address

A hierarchical approach to recognition of handwritten Bangla characters