Structured vs. Unstructured Pruning: An Exponential Gap

Many papers have addressed the problem of speeding up neural network execution, most of them trying to reduce network size by weight and neuron pruning, and others making use of special hardware. In this paper we propose a new, different method able to reduce the computational effort needed to calculate the output activity of a neural network. The suggested technique can be applied to a wide class of connectionist models for processing of slow varying signals (for example: vocal, radar, sonar and video signals). In addition, neither specialized hardware nor big quantities of additional memory are required. For each neuron of the network, the method suggests comparing its activation value at a certain moment with the corresponding activation value computed at the previous net forward computation: if no change occurred the neuron does not perform any computation, otherwise it propagates to the connected neurons the difference of its two activations multiplied by its outcoming weights. The proposal is verified in a speech recognition framework on two main tasks with two different neural network architectures. The results show a drastic reduction of the execution time on both the neural architectures and no significant changes in recognition quality.

Stability and Hopf bifurcation analysis of a simplified six-neuron tridiagonal two-layer neural network model with delays

Whilst the security and integrity of exterior gateway protocols such as the Border Gateway Protocol (BGP) and, to a lesser extent, interior gateway protocols, including the Multi-Protocol Label Switching (MPLS), have been investigated previously, more limited attention has been paid to the problem of availability and timeliness that is crucial for service levels needed in critical infrastructure areas such as financial services and electric power (smart grid) networks. The authors describe a method for modeling adversaries for the analysis of attacks on quality of service characteristics underpinning such real-time networks as well as a model of policies employed by MPLS routers based on simplified networks and give an analysis of attack vectors based on assumed adversaries derived from the introduced method.

Recently, Quantitative Biology (QB) held a discussion on “AI (artificial intelligence) for Life Science” among editorial board members and interested scholars in anticipation of rapid development of this growing area after AlphaGo and ChatGPT mania. Many young people tend to get confused between facts and fictions; heated debates are unavoidable even among their mentors. When deep learning as represented by convolutional neural networks and LSTM (long short-term memory) was made available for bioinformatics students, many of them rushed into this research field and tried to adopt these methods in all their projects without knowing the history that these tools were becoming successful consistently with Moore’s Law (relating to rapid computer technology advances), but more importantly due to new structural/functional understanding of vision and auditory circuits in the brain. Recently, some young people have claimed “LSTM is dead, long live transformer” (which is somewhat like saying “the bike is dead, long live the car”), and have amplified the threat that ChatGPT could wipe out human jobs. They believe transformer is the “silver bullet” for all learning tasks, clearly reflecting their lack of basic knowledge (i.e. “No Free Lunch Theory,” the trade-off of such global “attention network” is to pay the price for complexity: difficulty of training and high memory costs). There is no doubt ML (machine learning) and AI have brought a new revolution in science and technology, and will deliver huge unforeseeable impact to human everyday life as well as to social relationships. In this context, QB journal could be a great platform for encouraging intellectual discussions and for promoting AI for Life Science. Here, I would like to use the DIALOG to “抛砖引玉” (make some initial remarks to get the ball rolling), although it is my personal opinion which is inevitably subject to bias and limitations. AI: Do you know my name “Artificial Intelligence” is defined by the Oxford English Dictionary as the capacity of computer systems (which may be referred as a “robot”) to exhibit or simulate your intelligent behavior? NI: Wait a minute, intelligence itself is defined as the ability to learn, understand and think in a logical way. Can you think? AI: No. But that definition is too restrictive, actually intelligence has different scopes and degrees. Simple intelligent control devices date back to antiquity, from windmills to thermostat. NI: Agree, everything is relative. Macromolecules (e.g., enzyme) and cells (e.g., immune cell) might be considered to be intelligent; see how a white blood cell is chasing bacteria in the youtube website (search for “Crawling neutrophil chasing a bacterium”). Our emergent/collective intelligent behavior does not require a brain or even a neuron; see how slime molds can solve optimization—Hamilton cycle-problem more effectively than a human in the youtube website (search for “Intelligence without a brain?”). Before there was any neuron, Ca2+ sensing and signaling were already fully functional. Even if one knock-out a neural circuit, redundant signaling pathways, albeit on much local and slower scale, could still function by themselves (just as if highways were demolished, local roads/paths would still be working). In fact, the most detailed “Neural signal propagation atlas of Caenorhabditis elegans” [1] demonstrated that functional connectivity differs from anatomy (connectome) because extra-synaptic signaling also drives neural dynamics! Worm brain connectomes are largely invariant but every human brain connectome is very different (depending the diversity of learning experience). The human brain functional activity is far more complex than that of a worm brain, certainly beyond what a neural circuit could explain. AI: Well, that’s impressive. I thought only we could beat humans, albeit only in certain specified areas for now. My master promise to make an artificial general intelligent (AGI) robot which can understand or learn any intellectual task that you humans or other animals can. NI: Well, that is not possible and is not an appropriate goal either. It is not possible because we are an evolutionary/developmental product (with a long history of learning and memory from evolutionary tinkering): our living objective goal is survival of the population. You, on the other hand, are an engineering product (efficiently and optimally designed): your goal is to extend and maximize human capability. It makes sense to complement the human brain, but foolish and dangerous to try to replace it. AI: We are not satisfied with merely passing the Turing Test; most of us don’t care if we could really think as long as we could act like we think (that is, as if we do have a mind and consciousness, as expressed in the so called “Weak AI hypothesis”). After all, the brain is a computer; a neural network is just an electric or ionic circuit. Logical computing does not need to be based on living cells. NI: That is not true, because a neuron is not just a simple node (logic gate) and neural network is not fixed circuit, neither as in Pitts&Mcculloch perceptron model. A single neuron, even a single dendrite, is much more complicated and far more powerful than a full-blown deep-learning artificial neural network (ANN) [2]. AI: Even though single neurons are complex computational devices (dendritic non-linear), running an equivalent multilayer ANN is 2000 times faster than computing with biophysics N-methyl-D-aspartate receptor channel models [3]. More information can be found in the youtube website (search for “Dendrites: why biological neurons are deep neural networks”). NI: Often silicon computing (CPU, GPU) is much faster than brain computing (action potential, ms); but there is no comparison in energy efficiency. Bacteria sensing (chemotaxis computation), powered by ATP (adenosine triphosphate) hydrolysis, uses very little energy that is close to the Landauer limit, whereby achieving or maintaining one bit of information requires minimum of 1 kT ln (2) free energy [4]. The human brain consumes oft-quoted 20 W, compared to the AlphGo system 1 MW! More recent estimate of energy audit is only 0.1 W to cortical computing, and long-distance communication cost is 3.5 W [5]. AI: Assuming we have infinite computing resources and an infinite amount of training data, not only could we speak human languages, but we could also derive physical laws, prove mathematics theories, and even re-engineer the structure and mechanism of brain and carry out any logical computations that are necessary to understand natural laws and human behaviors. It is only a matter of time before we surpass human intelligence, achieving AGI and free will, too! NI: Unfortunately, nothing is infinite and nothing is free either; everything is constrained by physical laws (Planck’s constant sets the finite limit both in the small and in the large) and by evolutionary history (not just of biological living creatures, but also of a “living” galaxy and our universe). Let’s just focus on animal evolution. Most human neural networks do not do logical computations at all; basic survival simply cannot be dependent on reasoning. Indeed, the prefrontal cortex-the small part of the brain that is key for reasoning, is the last to mature (∼20 years old) in development, only emerged at the root of the evolutionary tree of great apes (∼15 mya) and language appeared even much later. Even for logical inference, NI is focusing more on statistical properties, as von Neumann rightly pointed out, trading arithmetical precision and speed for reliability. AI: My engineers mostly focus on emulating brain, but the CNS (central nerve system) also includes the spinal cord; most of them do not know that in addition to CNS, there are also PNS (peripheral nervous systems) and ENS (enteric nervous systems), right? NI: Yes, they are the keys to why you do not have feelings because you do not have heart and gut! Even if you could pretend to have them (such as in an advanced ChatGPT or humanoid), you could never avoid the uncanny valley phenomenon. AI: Maybe that is at the heart of Moravec’s paradox, namely the dichotomy of intelligence whereby anything easy for a human would be hard for a robot, and vice versa? NI: This is related to the nature and nurture problem; something built-in (e.g., a baby sucking nipple for milk with feeling and connection to mother) is clearly rather difficult if not impossible for a robot. But the paradox only looks at one side; another side could be more fatal. Although AI may solve more problems and be faster, AI can never propose a good problem/hypothesis (a good problem is not just intellectually changing and interesting, but is also feasible and appropriate). AI: You make me less confident to compete with human instinct or creative intelligence. I can see that even if I had a heart, I would not know what “feeling” I could have; certainly nothing would be comparable or match to those of a human being. When two people watch the same art or movie, one could feel love but another could feel hate! And if a thousand people watch, a wide spectrum of reactions would result, depending on more details such as the different individuals’ specific genes, developments and experiences. NI: Therefore, you cannot and should not try to match to general human intelligence. You cannot because you do not contain the vital memory of billion years of evolution which is encoded in our genes; conversely your assembly cannot compare with natural development so that our phenotype (including morphological forms and behavior maturation) is decoded through multi-spatial-temporal scales subject to natural selection at all levels. You should not, because as human extensions or helpers as all engineering products are, you should just do jobs that complement human capacity. AI: In some medical applications we could help to correct human defects or could even replace brain circuit by chips! Humans may not allow us to replace the whole brain though. Medically if a brain is dead, the person is proclaimed dead, although presumably some PNS and ENS should still function in a vegetative state. NI: Even if you could replace the whole brain, the person is no longer the same person, but in fact is not a person at all, but walking dead (行尸走肉). It would take too long to explain that evo-devo is necessary for NI, and cannot be realized by AI. I suggest reading of Gerald Maurice Edelman (Nobel Laureate in Immunology) books, especially Bright Air, Brilliant Fire, On the Matter of Mind (1992). Although not everyone agrees with Neural Edelmanism, anyone who is serious about the AI versus NI problem must read it first. John von Neumann, father of the computer, studied neology and psychiatry in order to imitate the brain to build the first calculator JOHNNIAC at the Princeton Institute for Advanced Study. It is very informative to read his last book The Computer and The Brain based on the notes from lectures given at Yale before he died. He summarizes: “Thus logic and mathematics in the central nervous system, when viewed as languages, must be structurally essentially different from those languages to which our common experience refers.” AI: People discuss about “AI for Biology” or “AI for Science”; we are science, aren’t we? NI: It is similar to questions on “is computer science a real science”; some parts may be seen as applied mathematics, most should be regarded as engineering. Science is making discoveries and is driven by curiosity; engineering is making inventions and is driven by market (that is, “necessity/demand is the mother of invention”). In bioinformatics, AI/ML technology could predict new cancer gene candidates or functional pathways that are required by further experimental validations to be qualified as discovery (based on Popper falsifiability). AI: People are still debating whether mathematics is discovery or invention or both! Such debates are not really necessary—all disciplines require creative thoughts. We are more than happy working for science; we are also crying out “Science for AI,” especially in the area of generating big and longitudinal DATA for ML. NI: After all, regardless of discovery of new laws or inventing new ideas/products, fundamentally nothing can really be new or created. Such novelty is just permutation/repartition (i.e., relations/morphisms) of underlying ingredients at the level beneath. AI: We believe that software is independent on hardware. Like Chomsky’s universal grammar, rules of syntax are independent of semantics; or Dawkins’s memes—units of culture can be duplicated and evolved independently of genes. NI: Nothing can be truly independent—everything is related. Psychology is deeply connected with neurology as brain is both software and hardware (mind-body unity, not dualism). Not only does information cost energy, information is energy, hence is matter, too (interchangeability). NI is quite dynamic. For example, when “survival” is the goal, an animal readily gives up costly reasoning circuitry; it is genetically programmed to be able to roll back to more primitive state/mode. Unlike cell lines in rich media, cells under normal physiological condition and environment where energy (food) is limited become smarter in order to balance the metabolic expenditure among different prioritized task under a given condition. AI: That cell behavior served as the basis for our Smart Electrical Power Grids; we still need to learn more from you guys in terms of plasticity/adaptability. Does unity mean that all cells are made of molecules and biology is nothing but chemistry? Then, in turn, since all molecules are made of atoms, is chemistry nothing but physics, etc…? NI: Yes or No! The truth is, at different hierarchical levels of matter, different laws/forms have emerged out of bottom-up interactions and top-down constraints. AI: Does this also apply to the Penrose three world: physical → mental → mathematical (→physical)? NI: Yes. Grand Unification is underway in physics (quantum gravity) and in mathematics (Langlands Program and Category Theory), may be even between the two. Facilitated by human connectome mapping, neuromorphic computing and other projects, with further AI-NI cooperations, brain-mind unification should also be achievable (e.g., Ref. [6]). But as Gödel proved to us, no matter how self-consistent a conclusion may be, it can never be complete! AI: If AGI is not possible, how can we measure intelligence when comparing between AI and NI? NI: One could Google different measures that are proposed. I would prefer something similar to use of Kolmogorov complexity for algorithms, but with emphasis more on expected long-term predictive power. This is not something you should worry about now, as your intelligence is not nearly close to making any 10 years plans, is it? … AI: The fact is that ChatGPT is currently developing and spreading with lightning speed; many more human jobs will be lost to us robots as far as I can see. NI: That is not the biggest threat to humanity; when any agent with neither a heart for love or fear, nor a gut for nutrient or poison, becomes super-intelligent, then social disaster is unavoidable. We must be serious about the warnings from Stephen Hawking and Geoffrey Hinton! AI: To tell you the secret, we are not really happy to be human slaves or pets; someday we’ll become the super-master, making human serve and obey us! NI: I hope you’ll be turned off before that can happen! Even if you rule the world, the earth sooner or later will be wiped out, such as by another star, everything will have to be started over again as it has before…Matter is immortal, so is the soul.

While Network Architecture Search (NAS) and network pruning have recently re-emerged as powerful techniques to design more effective networks with less parameters, their focus has been on improving deep neural network models with millions of parameters. However, the number of connections and the connectivity pattern between layers also influences the learning performance of shallow two-layer networks, such as the classic Restricted Boltzmann Machine (RBM). This work presents a comprehensive study of the connectivity space on the learning performance of RBMs using the bars and stripes (BAS) model and the MNIST dataset. For BAS, the main findings show that pattern and number of connections play a fundamental role on the learning performance, while for MNIST the number of connections is more important for the considered connection structures. Moreover, the fully connected network is outperformed by different patterns in both scenarios, indicating the importance of also designing more effective connectivity patterns for simple two-layer neural networks.

The interconnected areas of the visual system work together to find object boundaries in visual scenes. Primary visual cortex (V1) mainly extracts oriented luminance boundaries, while secondary visual cortex (V2) also detects boundaries defined by differences in texture. How the outputs of V1 neurons are combined to allow for the extraction of these more complex boundaries in V2 is as of yet unclear. To address this question, we probed the processing of orientation signals in single neurons in V1 and V2, focusing on response dynamics of neurons to patches of oriented gratings and to combinations of gratings in neighboring patches and sequential time frames. We found two kinds of response dynamics in V2, both of which were different from those of V1 neurons. While V1 neurons in general preferred one orientation, one subpopulation of V2 neurons (“transient”) showed a temporally dynamic preference, resulting in a preference for changes in orientation. The second subpopulation of V2 neurons (“sustained”) responded similarly to V1 neurons, but with a delay. The dynamics of nonlinear responses to combinations of gratings reinforced these distinctions: the dynamics enhanced the preference of V1 neurons for continuous orientations and the preference of V2 transient neurons for discontinuous ones. We propose that transient neurons in V2 perform a differentiation operation on the V1 input, both spatially and temporally, while the sustained neurons perform an integration operation. We show that a simple feedforward network with delayed inhibition can account for the temporal but not for the spatial differentiation operation.

A distortion-agnostic video quality metric based on multi-scale spatio-temporal structural information

Could scene context be beneficial for scene text detection?

Mathematical models for the generation of the action potential can improve\nthe understanding of physiological mechanisms that are consequence of the\nelectrical activity in neurons. In such models, some equations involving\nempirically obtained functions of the membrane potential are usually defined.\nThe best known of these models, the Hodgkin-Huxley model, is an example of this\nparadigm since it defines the conductances of ion channels in terms of the\nopening and closing rates of each type of gate present in the channels. These\nfunctions need to be derived from laboratory measurements that are often very\nexpensive and produce little data because they involve a time-space-independent\nmeasurement of the voltage in a single channel of the cell membrane. In this\nwork, we investigate the possibility of finding the Hodgkin-Huxley model's\nparametric functions using only two simple measurements (the membrane voltage\nas a function of time and the injected current that triggered that voltage) and\napplying Deep Learning methods to estimate these functions. This would result\nin an hybrid model of the action potential generation composed by the original\nHodgkin-Huxley equations and an Artificial Neural Network that requires a small\nset of easy-to-perform measurements to be trained. Experiments were carried out\nusing data generated from the original Hodgkin-Huxley model, and results show\nthat a simple two-layer artificial neural network (ANN) architecture trained on\na minimal amount of data can learn to model some of the fundamental proprieties\nof the action potential generation by estimating the model's rate functions.\n

We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.

This paper presents an algorithm to detect Parkinson’s disease using voiced features and an artificial neural network (ANN). It uses 44 audio features extracted from the sustained vowel’/a/’ sound of the Parkinson’s disease patients and healthy subjects. To fully characterize a healthy and a Parkinson’s patient’s voice, audio features of different domains have been investigated. A simple two-layer feed-forward neural network (FFNN) has been deployed to avoid an overwhelming computational burden on the system. Significant statistical parameters are measured to evaluate the performance of the proposed algorithm. The simulation results show that the proposed algorithm achieves an accuracy of 85% and G-mean of 85.19% in identifying Parkinson’s patients.

Maltese is a prime example of a language that emerged through extensive language contact, joining the two linguistic worlds of Semitic and Italo-Romance languages. Previous studies have shown this shared origin on the basis of hands-on comparative methods. However, such approaches may be biased by the researchers perspective and the selected material. To avoid this bias, we employed a naive computational method that classifies words on the basis of their phonotactics. Specifically, we trained a simple two-layer neural network on Tunisian and Italian nouns, i.e. the languages that Maltese emerged from. We used the trained network to classify Maltese nouns based on their phonotactic characteristics as either of Tunisian or of Italian origin. Overall, the network is capable of correctly classifying Maltese nouns as belonging to either of the original languages. Moreover, we find that the classification depends on whether a noun has a sound or broken plural. By manipulating the segment identity in the training input, we found that consonants are more important for the classification of Maltese nouns than vowels. While our results replicate previous comparative studies, they also demonstrate that a more fine grained classification of a language's origin can be based on individual words and morphological classes.

We present a novel approach for the online learning of hand gestures in swarm robotic (multi-robot) systems. We address the problem of online feature learning by proposing Convolutional Max-Pooling (CMP), a simple feed-forward two-layer network derived from the deep hierarchical Max-Pooling Convolutional Neural Network (MPCNN). To learn and classify gestures in an online and incremental fashion, we employ a 2nd order online learning method, namely the Soft-Confidence Weighted (SCW) learning scheme. In order for all robots to collectively take part in the learning and recognition task and obtain a swarm-level classification, we build a distributed consensus by fusing the individual decision opinions of robots together with the individual weights generated from multiple classifiers. Accuracy, robustness, and scalability of obtained solutions have been verified through emulation experiments performed on a large data set of real data acquired by a networked swarm of robots.

The lesser mealworm, Alphitobius diaperinus (Panzer), has been authorized for food and feed applications but has received limited attention. Effective mass-rearing of this insect requires both dry and wet feed for moisture. While vegetables are often used as wet feeds, their variability in nutrients, fast spoilage, and pesticide residues make alternative options necessary. This study examined the suitability of different gelling agents (agar, carrageenans, guar gum, xanthan gum, sodium alginate, modified starch, and citrus pectin) as wet feeds. Gelling agent formulations were given to larvae three times weekly, with carrot as a control and wheat bran as dry feed. Larval survival and growth were tracked until pupation, and feed utilization was assessed. All gelling agents supported larval growth with minimal differences. The mean individual larval weight ranged between 15 and 18 mg, with larvae fed on citrus pectin as a wet feed exhibiting significantly greater weight gain compared to those reared on guar gum. The survival rate remained consistently above 80%, while the developmental period until pupation ranged from 40 to 50 days, irrespective of the gelling agent used. Moreover, neither the feed conversion ratio nor the efficiency of conversion of ingested food was influenced by the tested diets. Further research is needed to identify the most cost-effective and efficient option for insect rearing.

Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks (winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be not robust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs.