Evaluating Embeddings using Syntax-based Classification Tasks as a Proxy for Parser Performance

Input and training signals are presented to recurrent artificial neural networks in an inherently sequential manner. However, of the two broad categories of practical problems to which recurrent neural networks are applied, signal prediction/reproduction and supervised signal classification, the information necessary to construct a training signal that can be presented over the full length of the input signal is usually only present in the first category. It is this absence of information in the second group that gives rise to differing problems, techniques and nomenclature in what could be otherwise be considered a subset of the signal prediction problem. The only information usually available for use as a training signal with classification tasks is the desired final classification of each training example. Normal practice is to present this piece of information, as a training signal of length one, concurrently with the last piece of input data, ignoring the possibility of providing a training signal prior to that point. The sequential nature of the training signal for classification tasks has been, to all intents and purposes, forgotten. This work explores the deficiency by examining the nature of suitable values, whether or not they can be constructed from the available information, and whether or not standard neural network training algorithms can be (or need to be) modified to make use of such values. It finds that methods do exist and develops a family of such methods, signal melding, which demonstrates the usefulness and possibilities inherent in filling in the missing values in the training signal. The signal melding technique makes use of the dynamic information seen at a network's outputs during the presentation of input. As a benefit, this information offers some important clues about the diversity of examples within an apparently homogeneous class. To make full use of these values, a modification to back propagation through time is developed, attentive back propagation through time, which shows clear advantages over the standard algorithm when using training signals over the full length of the input signal. Simulations taking over three years of run-time are presented which show that on simple classification tasks, when signal melding and attentive back propagation through time are combined, performance levels at least comparable with existing methods are achieved, together with much improved stability.

Dependency parsing, which is a fundamental task in Natural Language Processing (NLP), has attracted a lot of interest in recent years. In general, it is a module in an NLP pipeline together with word segmentation and Part-Of-Speech (POS) tagging in real Chinese NLP application. The NLP pipeline, which is a cascade system, will lead to error propagation for the parsing. This paper proposes a global discriminative re-ranking model using non-local features from word segmentation, POS tagging and dependency parsing to re-rank the parse trees produced by an N-best enhanced NLP pipeline. Experimental results indicate that the proposed model can improve the performance of dependency parsing as well as word segmentation and POS tagging in an NLP pipeline.

The goal of this work is to define a notion of a “quantum neural network” to classify data, which exploits the low-energy spectrum of a local Hamiltonian. As a concrete application, we build a binary classifier, train it on some actual data and then test its performance on a simple classification task. More specifically, we use Microsoft’s quantum simulator, LIQ[Formula: see text][Formula: see text], to construct local Hamiltonians that can encode trained classifier functions in their ground space, and which can be probed by measuring the overlap with test states corresponding to the data to be classified. To obtain such a classifier Hamiltonian, we further propose a training scheme based on quantum annealing which is completely closed-off to the environment and which does not depend on external measurements until the very end, avoiding unnecessary decoherence during the annealing procedure. For a network of size [Formula: see text], the trained network can be stored as a list of [Formula: see text] coupling strengths. We address the question of which interactions are most suitable for a given classification task, and develop a qubit-saving optimization for the training procedure on a simulated annealing device. Furthermore, a small neural network to classify colors into red versus blue is trained and tested, and benchmarked against the annealing parameters.

Natural language processing is the area of study that focuses on how computers and human languages interact. Machine translation, sentiment analysis, semantic analysis, and text analysis are a few of them. The key natural language processing component is morphological analysis, which breaks words into their corresponding morphemes to determine their structure and meaning. Dependency parsing algorithms use morphological information to determine the syntactic structure of a sentence. This study evaluates the performance of various parsers, including Turbo parser, Lys-FASTPARSE, UU parser, and neural network based parser, to analyse dependency parsing methodologies used in the Malayalam language. The study evaluates the performance of these parsers in handling the difficulties and effectiveness of extensive morphological and syntactic features of Malayalam. Among these parsers, Lys-FASTPARSE performs better in LAS F1 score, MLAS score, and BLEX score, maintaining values of 56.60 and 48.58 before and after optimization. The neural network parser shows minor improvements in unlabelled attachment scores from 0.72 to 0.73 and labelled attachment scores from 0.46 to 0.47. With an LAS of 66.89% and UAS of 87.12%, the Turbo parser shows better results for baseline performance. The precision of 98.81% and recall of 88.42% in binned HEAD directions of the UU parser shows its performance in managing right direction dependencies. While lower, the parser's performance in managing left-direction and root dependencies still reflects its ability to navigate complex syntactic structures effectively. The results underscore the significance of tailored parsing techniques for morphologically rich languages like Malayalam and provide insights into optimizing parser performance for improved syntactic analysis.

Dependency parsing in Myanmar face many challenges due to agglutinative morphology, flexible word order, and lack of explicit word boundaries. This paper addresses these challenges by investigating the performance of Myanmar dependency parsing based three pretrained-transformer models: XLM-RoBERTa-Large, XLM-RoBERTa-base-Longformer-4096, and language-specific model, MyanBERTa. The proposed methodology utilizes a parameter-efficient model with adapter layers and a biaffine parsing mechanism by optimizing multi-task learning objective across six linguistic prediction tasks. The experimental results demonstrated that XLM-RoBERTa-Large achieves the highest Labeled Attachment Score (LAS) and Unlabeled Attachment Score (UAS) on both development set and test set. Despite XLM-RoBERTa-base-Longformer-4096 model can extended its input capacity and MyanBERTa is pretrained on a language capacity, their performance drop compared to XLM-RoBERTa-Large. These findings suggest that the scale and architectural strength of models are more critical for high-performance dependency parsing than models with an extended input capacity or those trained on a language-specific corpus.

Violence detection in videos is very promising in practical applications due to the emergence of massive videos in recent years. Most previous works define violence detection as a simple video classification task and use the single modality of small-scale datasets, e.g., visual signal. However, such solutions are undersupplied. To mitigate this problem, we study weakly supervised violence detection on the large-scale audio-visual violence data, and first introduce two complementary tasks, i.e., coarse-grained violent frame detection and fine-grained violent event detection, to advance the simple violence video classification to frame-level violent event localization, which aims to accurately locate the violent events on untrimmed videos. We then propose a novel network that takes as input audio-visual data and contains three parallel branches to capture different relationships among video snippets and further integrate features, where similarity branch and proximity branch capture long-range dependencies using similarity prior and proximity prior, respectively, and score branch dynamically captures the closeness of predicted score. In both coarse-grained and fine-grained tasks, our approach outperforms other state-of-the-art approaches on two public datasets. Moreover, experiment results also show the positive effect of audio-visual input and relationship modeling.

Towards Spike-Based Speech Processing: A Biologically Plausible Approach to Simple Acoustic ClassificationShortcomings of automatic speech recognition (ASR) applications are becoming more evident as they are more widely used in real life. The inherent non-stationarity associated with the timing of speech signals as well as the dynamical changes in the environment make the ensuing analysis and recognition extremely difficult. Researchers often turn to biology seeking clues to make better engineered systems, and ASR is no exception with the usage of feature sets such as Mel frequency cepstral coefficients, which employ filter banks similar to cochlear filter banks in frequency distribution and bandwidth. In this paper, we delve deeper into the mechanics of the human auditory system to take this biological inspiration to the next level. The main goal of this research is to investigate the computation potential of spike trains produced at the early stages of the auditory system for a simple acoustic classification task. First, various spike coding schemes from temporal to rate coding are explored, together with various spike-based encoders with various simplicity levels such as rank order coding and liquid state machine. Based on these findings, a biologically plausible system architecture is proposed for the recognition of phonetically simple acoustic signals which makes exclusive use of spikes for computation. The performance tests show superior performance on a noisy vowel data set when compared with a conventional ASR system.

While the predictive performance of modern statistical dependency parsers relies heavily on the availability of expensive expert-annotated treebank data, not all annotations contribute equally to the training of the parsers. In this paper, we attempt to reduce the number of labeled examples needed to train a strong dependency parser using batch active learning (AL). In particular, we investigate whether enforcing diversity in the sampled batches, using determinantal point processes (DPPs), can improve over their diversity-agnostic counterparts. Simulation experiments on an English newswire corpus show that selecting diverse batches with DPPs is superior to strong selection strategies that do not enforce batch diversity, especially during the initial stages of the learning process. Additionally, our diversity-aware strategy is robust under a corpus duplication setting, where diversity-agnostic sampling strategies exhibit significant degradation.

In the past few years, performance of dependency parsing has been improved by large margin on closed-domain benchmark datasets. However, when processing real-life texts, parsing performance degrades dramatically. Besides the domain adaptation technique, which has made slow progress due to its intrinsic difficulty, one straightforward way is to annotate a certain scale of syntactic data given a new source of texts. However, it is well known that annotating data is time and effort consuming, especially for the complex syntactic annotation. Inspired by the progress in crowdsourcing, this paper proposes to annotate noisy multi-annotation syntactic data with non-experts annotators. Each sentence is independently annotated by multiple annotators and the inconsistencies are retained. In this way, we can annotate data very rapidly since we can recruit many ordinary annotators. Then we construct and release three multi-annotation datasets from different sources. Finally, we propose and compare several benchmark approaches to training dependency parsers on such multi-annotation data. We will release our code and data at http://hlt.suda.edu.cn/~zhli/.

Defect recognition is an important task in intelligent manufacturing. Due to the subjectivity of human annotation, the collected defect data usually contains a lot of noise and unpredictable uncertainties, which have a great negative influence on defect recognition. It is a significant challenge to discover an effective defect recognition model with satisfactory uncertainty processing ability. A natural way is to automatically search for an efficient deep model, which can be realized by neural architecture search (NAS). To achieve this, we propose an efficient fuzzy NAS framework for defect recognition, where the searched architecture can effectively handle uncertain information from the given datasets. Specifically, we first design a fuzzy search space and the related encoding strategy for fuzzy NAS. Then, we propose a comparator-based evolutionary search approach, where an online end-to-end comparator is learned to directly determine the selection of candidate architectures from the evolutionary population. The comparator works in an end-to-end way and it transforms the complex ranking problem of evaluating architectures into a simple classification task, which overcomes the rank disorder issue suffered from traditional performance predictors. A series of experimental results demonstrate that the architecture with fewer #Params (1.22 M) search by fuzzy neural architecture search framework for defect recognition method achieves higher accuracy (92.26%) compared to the state-of-the-art results (i.e., DARTS-PV) on the ELPV dataset, as well as competitive results (accuracy = 76.4%, #Params = 1.04 M) on the CODEBRIM dataset. Experimental results show the effectiveness and efficiency of our proposed method in handling uncertain problems.

This work presents an approach to delay-based reservoir computing (RC) at the sensor level without input modulation. It employs a time-multiplexed bias to maintain transience while utilizing either an electrical signal or an environmental signal (such as acceleration) as an unmodulated input signal. The proposed approach enables RC carried out by sufficiently nonlinear sensory elements, as we demonstrate using a single electrostatically actuated microelectromechanical system (MEMS) device. The MEMS sensor can perform colocalized sensing and computing with fewer electronics than traditional RC elements at the RC input (such as analog-to-digital and digital-to-analog converters). The performance of the MEMS RC is evaluated experimentally using a simple classification task, in which the MEMS device differentiates between the profiles of two signal waveforms. The signal waveforms are chosen to be either electrical waveforms or acceleration waveforms. The classification accuracy of the presented MEMS RC scheme is found to be over 99%. Furthermore, the scheme is found to enable flexible virtual node probing rates, allowing for up to 4× slower probing rates, which relaxes the requirements on the system for reservoir signal sampling. Finally, our experiments show a noise-resistance capability for our MEMS RC scheme.

Reservoir computing is a recent approach from the fields of machine learning and artificial neural networks to solve a broad class of complex classification and recognition problems such as speech and image recognition. As is typical for methods from these fields, it involves systems that were trained based on examples, instead of using an algorithmic approach. It originated as a new training technique for recurrent neural networks where the network is split in a reservoir that does the `computation' and a simple readout function. This technique has been among the state-of-the-art. So far implementations have been mainly software based, but a hardware implementation offers the promise of being low-power and fast. We previously demonstrated with simulations that a network of coupled semiconductor optical amplifiers could also be used for this purpose on a simple classification task. This paper discusses two new developments. First of all, we identified the delay in between the nodes as the most important design parameter using an amplifier reservoir on an isolated digit recognition task and show that when optimized and combined with coherence it even yields better results than classical hyperbolic tangent reservoirs. Second we will discuss the recent advances in photonic reservoir computing with the use of resonator structures such as photonic crystal cavities and ring resonators. Using a network of resonators, feedback of the output to the network, and an appropriate learning rule, periodic signals can be generated in the optical domain. With the right parameters, these resonant structures can also exhibit spiking behaviour.

This study aims to evaluate the performance of key Federated Learning (FL) algorithms—FedSGD(Federated Stochastic Gradient Descent), FedAVG(Federated Averaging), and FedRep(Federated Representation Learning)—with a view toward future expansion into vision-based multitask learning. Federated learning, a distributed training paradigm that prioritizes data privacy and communication efficiency, faces significant performance degradation in real-world settings due to the frequent presence of Non-IID (non-identically distributed) data across clients. In this work, we partitioned the FEMNIST(Federated Extended MNIST) dataset in a Non-IID manner and quantitatively assessed the convergence speed, accuracy, and communication efficiency of each algorithm. Experimental results show that FedAVG consistently delivers strong performance and stability, while FedRep demonstrates advantages in personalized learning. FedSGD, despite requiring fewer communications, exhibited notable limitations in accuracy. These findings provide valuable insights for selecting appropriate algorithms when expanding FL to multitask scenarios, such as object detection and semantic segmentation, beyond simple classification tasks.

The ultimate goal of artificial intelligence (AI) is to mimic the human brain to perform decision-making and control directly from high-dimensional sensory input. Diffractive optical networks (DONs) provide a promising solution for implementing AI with high speed and low power-consumption. Most reported DONs focus on tasks that do not involve environmental interaction, such as object recognition and image classification. By contrast, the networks capable of decision-making and control have not been developed. Here, we propose using deep reinforcement learning to implement DONs that imitate human-level decision-making and control capability. Such networks, which take advantage of a residual architecture, allow finding optimal control policies through interaction with the environment and can be readily implemented with existing optical devices. The superior performance is verified using three types of classic games: tic-tac-toe, Super Mario Bros., and Car Racing. Finally, we present an experimental demonstration of playing tic-tac-toe using the network based on a spatial light modulator. Our work represents a solid step forward in advancing DONs, which promises a fundamental shift from simple recognition or classification tasks to the high-level sensory capability of AI. It may find exciting applications in autonomous driving, intelligent robots, and intelligent manufacturing.

Evolutionary neural networks, or neuroevolution, appear to be a promising way to build versatile adaptive systems, combining evolution and learning. One of the most challenging problems of neuroevolution is finding a scalable and robust genetic representation, which would allow to effectively grow increasingly complex networks for increasingly complex tasks. In this paper we propose a novel developmental encoding for networks, featuring scalability, modularity, regularity and hierarchy. The encoding allows to represent structural regularities of networks and build them from encapsulated and possibly reused subnetworks. These capabilities are demonstrated on several test problems. In particular for parity and symmetry problems we evolve solutions, which are fully general with respect to the number of inputs. We also evolve scalable and modular weightless recurrent networks capable of autonomous learning in a simple generic classification task. The encoding is very flexible and we demonstrate this by evolving networks capable of learning via neuromodulation. Finally, we evolve modular solutions to the retina problem, for which another well known neuroevolution method—HyperNEAT—was previously shown to fail. The proposed encoding outperformed HyperNEAT and Cellular Encoding also in another experiment, in which certain connectivity patterns must be discovered between layers. Therefore we conclude the proposed encoding is an interesting and competitive approach to evolve networks.