The Tsetlin Machine - From Arithmetic to Logic-based AI

Tsetlin Machine (TM) is an interpretable pattern recognition algorithm based on propositional logic, which has demonstrated competitive performance in many Natural Language Processing (NLP) tasks, including sentiment analysis, text classification, and Word Sense Disambiguation. To obtain human-level interpretability, legacy TM employs Boolean input features such as bag-of-words (BOW). However, the BOW representation makes it difficult to use any pre-trained information, for instance, word2vec and GloVe word representations. This restriction has constrained the performance of TM compared to deep neural networks (DNNs) in NLP. To reduce the performance gap, in this paper, we propose a novel way of using pre-trained word representations for TM. The approach significantly enhances the performance and interpretability of TM. We achieve this by extracting semantically related words from pre-trained word representations as input features to the TM. Our experiments show that the accuracy of the proposed approach is significantly higher than the previous BOW-based TM, reaching the level of DNN-based models.

Network-based anomaly detection and prevention systems are crucial to an organization’s success. In recent years, many rule-based and machine learning-based anomaly detection and prevention approaches have been explored to ensure the security and reliability of computer networks within enterprises. Although these ML-based approaches worked well, they are not transparent and do not interpret the results. Recently, the Tsetlin Machine (TM), a rule-based machine learning model with interpretability, has gained considerable attention within a wide range of applications. In this work, we propose a methodology for a network-based anomaly detection system that uses TM. We considered the UNSW-NB15 dataset with 9 anomaly classes for efficient classification and interpretation. By integrating KBins discretization with methods like SMOTE, Borderline-SMOTE, SVMSMOTE, and ADASYN, we address class imbalance and enhance the detection of rare attacks. The experiments show a higher accuracy of 96.89% with TM outperforming the other machine learning models. In addition, we highlight the importance of SMOTE techniques in enhancing TM performance. Additionally, we present the interpretation results to identify the features that contribute to anomaly detection.

Advances in QSAR are led by two core paradigms, 1) descriptor engineering, where complex fixedlength vectors of compounds are generated and conventional ML methods are applied to those representations and 2) raw chemical inputs (e.g., SMILES, 2D-graph) being provided to deep learning neural network models, which construct their own internal representations of molecules and learn iteratively over them.Here we present the Tsetlin Machine (TM) -which combines the accuracy and easy-use of existing rule-based QSAR ML methods (e.g., RF and XGBoost), the iterative learning aspect of NN algorithms and its intrinsic interpretability.The TM uses teams of finite-state automata which capture frequent patterns as propositional logic (clauses) via reinforcement learning.The benchmarking pipeline presented here demonstrates that TM-QSAR coupled with ECFP4 descriptors frequently performs better than existing rule-based QSAR methods for ROC-AUC, PRC-AUC and PPV, with a high capacity for inter-scaffold generalisation.However, due to the binary nature of TM-QSAR, performance it is currently limited when descretised continuous descriptors are used.TM-QSAR demonstrated particularly impressive classification scores for MOR (ROC-AUC = 0.87, PRC-AUC = 0.77) and CYPA4 (ROC-AUC = 0.92, PRC-AUC = 0.63), when compared to RF and XGBoost.Using TM in combination with substructural fingerprinting descriptors allows for an interpretability suite which can be extracted directly from clauses.Here we detail molecule property maps (TM-MPM) to view atom-wise TM-QSAR bioactivity contributions for single molecules and closed-form WAC scores (Weights Activations Clauses) for descriptor-wise contributions to regions of predicted chemical space.These methods show strong alignment of TM-QSAR interpretations to known ligandprotein interactions of the MOR target and gives non-linear, conditional interpretations for greater predicted bioactivity.Given this combination of accuracy, computational efficiency and interpretability, we provide a basis for TM-QSAR to be explored as a standard methodology in virtual screening toolkits.

We present a hardware design for the learning datapath of the Tsetlin machine algorithm, along with a latency analysis of the inference datapath. In order to generate a low energy hardware which is suitable for pervasive artificial intelligence applications, we use a mixture of asynchronous design techniques—including Petri nets, signal transition graphs, dualrail and bundled-data. The work builds on previous design of the inference hardware, and includes an in-depth breakdown of the automaton feedback, probability generation and Tsetlin automata. Results illustrate the advantages of asynchronous design in applications such as personalized healthcare and battery-powered internet of things devices, where energy is limited and latency is an important figure of merit. Challenges of static timing analysis in asynchronous circuits are also addressed.

Building models that are both interpretable and accurate is an unresolved challenge for many pattern recognition problems. In general, rule-based and linear models lack accuracy, while deep learning interpretability is based on rough approximations of the underlying inference. However, recently, the rule-based Tsetlin Machines (TMs) have obtained competitive performance in terms of accuracy, memory footprint, and inference speed on diverse benchmarks (image classification, regression, natural language understanding, and game-playing). TMs construct rules using human-interpretable conjunctive clauses in propositional logic. These, in turn, are combined linearly to solve complex pattern recognition tasks. This paper addresses the accuracy-interpretability challenge in machine learning by introducing a TM with integer weighted clauses - the Integer Weighted TM (IWTM). The intent is to increase TM interpretability by reducing the number of clauses required for competitive performance. The IWTM achieves this by weighting the clauses so that a single clause can replace multiple duplicates. Since each TM clause is formed adaptively by a Tsetlin Automata (TA) team, identifying effective weights becomes a challenging online learning problem. We solve this problem by extending each team of TA with another kind of automaton: the stochastic searching on the line (SSL) automaton. We evaluate the performance of the new scheme empirically using five datasets, along with a study of interpretability. On average, IWTM uses 6.5 times fewer literals than the vanilla TM and 120 times fewer literals than a TM with real-valued weights. Furthermore, in terms of average memory usage and F1-Score, IWTM outperforms simple Multi-Layered Artificial Neural Networks, Decision Trees, Support Vector Machines, K-Nearest Neighbor, Random Forest, Gradient Boosted Trees (XGBoost), Explainable Boosting Machines (EBMs), as well as the standard and real-value weighted TMs. IWTM finally outperforms Neural Additive Models on Fraud Detection and StructureBoost on CA-58 in terms of Area Under Curve, while performing competitively on COMPAS.

In the paper, an approach to the detection of cyber-attacks is proposed that involves detecting anomalous network traffic using the Tsetlin machine. The experimental studies carried out for different types of network attacks have demonstrated the efficiency of this proposed approach.

Tsetlin machines (TMs) are a pattern recognition approach that uses finite state machines for learning and propositional logic to represent patterns. In addition to being natively interpretable, they have provided competitive accuracy for various tasks. In this paper, we increase the computing power of TMs by proposing a first-order logic-based framework with Herbrand semantics. The resulting TM is relational and can take advantage of logical structures appearing in natural language, to learn rules that represent how actions and consequences are related in the real world. The outcome is a logic program of Horn clauses, bringing in a structured view of unstructured data. In closed-domain question-answering, the first-order representation produces 10 × more compact KBs, along with an increase in answering accuracy from 94.83% to 99.48%. The approach is further robust towards erroneous, missing, and superfluous information, distilling the aspects of a text that are important for real-world understanding

A set of variables is the Markov blanket of a random variable if it contains all the information needed for predicting the variable. If the blanket cannot be reduced without losing useful information, it is called a Markov boundary. Identifying the Markov boundary of a random variable is advantageous because all variables outside the boundary are superfluous. Hence, the Markov boundary provides an optimal feature set. However, learning the Markov boundary from data is challenging for two reasons. If one or more variables are removed from the Markov boundary, variables outside the boundary may start providing information. Conversely, variables within the boundary may stop providing information. The true role of each candidate variable is only manifesting when the Markov boundary has been identified. In this paper, we propose a new Tsetlin Machine (TM) feedback scheme that supplements Type I and Type II feedback. The scheme introduces a novel Finite State Automaton — a Context-Specific Independence Automaton. The automaton learns which features are outside the Markov boundary of the target, allowing them to be pruned from the TM during learning. We investigate the new scheme empirically, showing how it is capable of exploiting context-specific independence to find Markov boundaries. Further, we provide a theoretical analysis of convergence. Our approach thus connects the field of Bayesian networks (BN) with TMs, potentially opening up for synergies when it comes to inference and learning, including TM-produced Bayesian knowledge bases and TM-based Bayesian inference.