MeTHanol: Modularized Thinking Language Models with Intermediate Layer Thinking, Decoding and Bootstrapping Reasoning

BackgroundMedical entity recognition is a key technology that supports the development of smart medicine. Existing methods on English medical entity recognition have undergone great development, but their progress in the Chinese language has been slow. Because of limitations due to the complexity of the Chinese language and annotated corpora, these methods are based on simple neural networks, which cannot effectively extract the deep semantic representations of electronic medical records (EMRs) and be used on the scarce medical corpora. We thus developed a new Chinese EMR (CEMR) dataset with six types of entities and proposed a multi-level representation learning model based on Bidirectional Encoder Representation from Transformers (BERT) for Chinese medical entity recognition.ObjectiveThis study aimed to improve the performance of the language model by having it learn multi-level representation and recognize Chinese medical entities.MethodsIn this paper, the pretraining language representation model was investigated; utilizing information not only from the final layer but from intermediate layers was found to affect the performance of the Chinese medical entity recognition task. Therefore, we proposed a multi-level representation learning model for entity recognition in Chinese EMRs. Specifically, we first used the BERT language model to extract semantic representations. Then, the multi-head attention mechanism was leveraged to automatically extract deeper semantic information from each layer. Finally, semantic representations from multi-level representation extraction were utilized as the final semantic context embedding for each token and we used softmax to predict the entity tags.ResultsThe best F1 score reached by the experiment was 82.11% when using the CEMR dataset, and the F1 score when using the CCKS (China Conference on Knowledge Graph and Semantic Computing) 2018 benchmark dataset further increased to 83.18%. Various comparative experiments showed that our proposed method outperforms methods from previous work and performs as a new state-of-the-art method.ConclusionsThe multi-level representation learning model is proposed as a method to perform the Chinese EMRs entity recognition task. Experiments on two clinical datasets demonstrate the usefulness of using the multi-head attention mechanism to extract multi-level representation as part of the language model.

Existing language model compression methods mostly use a simple L_2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. Although widely used, this objective by design assumes that all the dimensions of hidden representations are independent, failing to capture important structural knowledge in the intermediate layers of the teacher network. To achieve better distillation efficacy, we propose Contrastive Distillation on Intermediate Representations (CoDIR), a principled knowledge distillation framework where the student is trained to distill knowledge through intermediate layers of the teacher via a contrastive objective. By learning to distinguish positive sample from a large set of negative samples, CoDIR facilitates the student's exploitation of rich information in teacher's hidden layers. CoDIR can be readily applied to compress large-scale language models in both pre-training and finetuning stages, and achieves superb performance on the GLUE benchmark, outperforming state-of-the-art compression methods.

Large Language Models (LLMs) are positioned as generalist models often claiming superlative performance on many Natural Language Processing (NLP) tasks.However, they tend to fail at Quality Estimation (QE) of Machine Translation (MT), particularly for low-resource languages.The talk investigates root causes of this disparity, such as tokenization inconsistencies arising from morphological richness in natural languages.To bridge this gap, the talk introduces strategies to embed annotation guidelines-based reasoning constraints directly in-context.Furthermore, our investigation on optimal cross-lingual alignment shows that intermediate Transformer layers help produce performant adapters.By attaching Low-Rank Adapter (LoRA) based regression heads to intermediate layers, we bypass the generation-specific biases of the final layer, efficiently outperforming standard instruction fine-tuning and SoTA encoders like COMETKiwi.Finally, via results from the WMT Unified Shared subtask on QE-informed Correction, we demonstrate that these precise estimations can guide LLMs to produce reliable corrections.We discuss how these signals help address the diminishing returnschallenge, enabling models to improve fluent outputs without diverging from human references. Bio:Researcher working on problems within areas of Natural Language Processing (NLP) and Machine Learning (ML) at the Institute for People-Centred AI (PAI) and School of Computer Science and Electronic Engineering.As a research lead, I manage the NLP subgroup within the Nature Inspired Computing and Engineering group (NICE) @ Computer Science Research Centre.I also lead teaching on the NLP module offered to both undergraduate and postgraduate students.My research focuses on developing scalable and safe human-machine interaction using foundation models.Guided by the principles of Responsible and Inclusive AI, my work emphasises cross-lingual and multimodal representation learning to address challenges like online toxicity, misinformation, and digital accessibility for low-resource languages.Our research outcomes-code, data, and models, are publicly available on the SurreyNLP GitHub and HuggingFace.

Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.

Existing IR measures for offline evaluation directly bring in the labels into computation, where the labels are on the entire documents. This direct dependency makes the measure highly reliant on the completeness of the labels, consequently the measure values are sensitive towards missing labels, resulting in poor robustness and reusability. To mitigate this, we propose a novel evaluation approach, constructing an intermediate layer between the labels and the measure, improving the robustness and reusability by dampening the direct dependency, as well as considering the content of the document in the measure computation. In particular, we propose to estimate a language model based on a selected relevant document set to construct a ground truth, afterward using the divergence between the search result and this ground truth to compute measures. To further save labeling efforts and to improve efficiency, we select representative documents, query set and topic terms involved in the evaluation separately before computing the measure. Preliminary experiments on the diversity tasks of TREC Web Track 2009-2012, using ClueWeb09-A as a document collection, show that with as little as 30% of judgments our novel approach almost accurately reconstructs the original system rankings determined by α-nDCG, ERR-IA, and NRBP.

Large language models (LLMs) have revolutionized natural language processing (NLP) by excelling at understanding and generating human-like text.However, their widespread deployment can be prohibitively expensive.Sort-edNet is a recent training technique for enabling dynamic inference by leveraging the modularity in networks and sorting sub-models based on computation/accuracy in a nested manner.We extend SortedNet to generative NLP tasks, making large language models dynamic without any Pre-Training and by only replacing Standard Fine-Tuning (SFT) with Sorted Fine-Tuning (SoFT).Our approach boosts model efficiency, eliminating the need for multiple models for various scenarios during inference.We show that this approach can unlock the potential of intermediate layers of transformers in generating the target output.Our sub-models remain integral components of the original model, minimizing storage requirements and transition costs between different computational/latency budgets.The efficacy of our proposed method was demonstrated by applying it to tune LLaMA 2 13B on the Stanford Alpaca dataset for instruction following and TriviaQA for closed-book question answering.Our results show the superior performance of sub-models in comparison to Standard Fine-Tuning and SFT+ICT (Early-Exit), all achieved with very efficient tuning and without additional memory usage during inference.1 github.com/WeOpenML/PandaLM/blob/main/data/testsetinference-v1.jsoning multiple models simultaneously as explained in the previous section.