Large Language Models for Effective Detection of Algorithmically Generated Domains: A Comprehensive Review

Malware families often use the Domain Generation Algorithm (DGA) to communicate with the Command and Control (C&C) servers. Although machine learning and deep learning based methods have achieved good accuracy in DGA detection task, it has problems on the new DGA families with limited datasets. In this paper, RL-Gen, a Reinforcement Learning (RL) framework, is proposed to improve the performance of character-level text generation with few input samples. RL-Gen has two modules, W-Generator and Evaluator. W-Generator is an improved generation model based on WGAN-GP, which is regarded as an agent, and Evaluator acts as an environment to evaluate the generated text. Especially in DGA case, Evaluator is an effective DGA detection model (ATT-GRU). The parameters’ updating of W-Generator is optimized by the reward from Evaluator, which promotes the generating abilities on speed and quality. Experiments show that the generated DGAs are sufficiently close to real DGAs, and can play an alternative role of real DGAs in detection model training. And RL-Gen gets better quality of text generation more quickly and smoothly than WGAN-GP.

Domain generation algorithms (DGAs) automatically generate large numbers of domain names in DNS domain fluxing for the purpose of command-and-control (C&C) communication. DGAs are immune to static prevention methods like blacklisting and sinkholing. Detection of DGAs in a live stream of queries in a DNS server is referred to as inline detection. Most of the previous approaches in the literature on DGA detection either: (i) are based on small synthetic data sets for training, rather than data collected from real traffic or (ii) require contextual information and therefore cannot be used for inline detection. In this work, we overcome these limitations by proposing a novel way to label a large volume of data collected from real traffic as DGA/non-DGA and by using deep learning techniques. Our classifiers can be trained with large amounts of real traffic, rather than small synthetic data sets, and therefore have better performance.

DGA-based botnet, which uses Domain Generation Algorithms (DGAs) to evade supervision, has become a part of the most destructive threats to network security. Over the past decades, a wealth of defense mechanisms focusing on domain features have emerged to address the problem. Nonetheless, DGA detection remains a daunting and challenging task due to the big data nature of Internet traffic and the potential fact that the linguistic features extracted only from the domain names are insufficient and the enemies could easily forge them to disturb detection. In this paper, we propose a novel DGA detection system which employs an incremental word-embeddings method to capture the interactions between end hosts and domains, characterize time-series patterns of DNS queries for each IP address and therefore explore temporal similarities between domains. We carefully modify the Word2Vec algorithm and leverage it to automatically learn dynamic and discriminative feature representations for over 1.9 million domains, and develop an simple classifier for distinguishing malicious domains from the benign. Given the ability to identify temporal patterns of domains and update models incrementally, the proposed scheme makes the progress towards adapting to the changing and evolving strategies of DGA domains. Our system is evaluated and compared with the state-of-art system FANCI and two deep-learning methods CNN and LSTM, with data from a large university’s network named TUNET. The results suggest that our system outperforms the strong competitors by a large margin on multiple metrics and meanwhile achieves a remarkable speed-up on model updating.

In order to stay undetected and keep their operations alive, cyber criminals are continuously evolving their methods to stay ahead of current best defense practices. Over the past decade, botnets have developed from using statically hardcoded IP addresses and domain names to randomly-generated ones, so-called domain generation algorithms (DGA). Malicious software coordinated via DGAs leaves however a distinctive signature in network traces of high entropy domain names, and a variety of algorithms have been introduced to detect certain aspects about currently used DGAs.

The article addresses the problem of detecting domains generated by Domain Generation Algorithms (DGA) which are widely used by attackers to build robust botnet control channels and covert communication. Traditional methods are based on manual feature engineering or specialized neural network architectures that reduce their robustness to evolving DGA families. The scientific novelty of the proposed approach lies in the use of Large Language Models (LLM) by leveraging their contextual adaptation mechanism to identify hidden patterns in domain names and classify them. The developed approach is based on the use of LLMs which receives examples of legitimate and generated domains within the context. To improve the efficiency, example selection strategies (TopK, VoteK), various metrics of data homogeneity and variability are used. Additionally, the influence of the domain name length and entropy on the stability of the approach is analyzed. The experimental part is performed on a dataset including 68 DGA families and a subset of legitimate Tranco domains. The training set included 54 families, and testing took place on all 68 families, including previously unseen 14 families. Results showed the efficiency of the approach: precision = 0.93, recall = 0.95 and F1-measure = 0.94. The ability of LLM to generalize rules to new DGA families is confirmed. Compared with existing methods, the proposed approach does not require additional retraining and provides flexibility due to contextual adaptation. It demonstrated resistance to noise and the capability to detect new DGA families, which makes its application promising in the field of cybersecurity. At the same time, the sensitivity of the model to the length of domain names and the need for context balancing were revealed. Promising areas of development are the integration of additional features (DNS metadata, query time series) and methods for adaptation to stream processing.

As the most commonly used attack strategy by Botnets, the Domain Generation Algorithm (DGA) has strong invisibility and variability. Using deep learning models to detect different families of DGA domain names can improve the network defense ability against hackers. However, this task faces an extremely imbalanced sample size among different DGA categories, which leads to low classification accuracy for small sample categories and even classification failure for some categories. To address this issue, we introduce the long-tailed concept and augment the data of small sample categories by transferring pre-trained knowledge. Firstly, we propose the Data Balanced Review Method (DBRM) to reduce the sample size difference between the categories, thus a relatively balanced dataset for transfer learning is generated. Secondly, we propose the Knowledge Transfer Model (KTM) to enhance the knowledge of the small sample categories. KTM uses a multi-stage transfer to transfer weights from the big sample categories to the small sample categories. Furthermore, we propose the Knowledge Distillation Transfer Model (KDTM) to relieve the catastrophic forgetting problem caused by transfer learning, which adds knowledge distillation loss based on the KTM. The experimental results show that KDTM can significantly improve the classification performance of all categories, especially the small sample categories. It can achieve a state-of-the-art macro average F1 score of 84.5%. The robustness of the KDTM model is verified using three DGA datasets that follow the Pareto distributions.

In today's cyber environment, modern botnets and malware are increasingly employing domain generation mechanisms to circumvent conventional detection solutions reliant on blacklisting or statistical methods for malicious domains. These outdated methods prove inadequate against algorithmically generated domain names, presenting significant challenges for cyber security. Domain Generation Algorithms (DGAs) have become essential tools for many malware families, allowing them to create numerous DGA domain names to establish communication with C&C servers. Consequently, detecting such malware has become a formidable task in cyber security. Traditional approaches to domain name detection rely heavily on manual feature engineering and statistical analysis, with classifiers designed to differentiate between legitimate and DGA domain names. In this study, we propose a novel approach to classify and detect algorithmically generated domain names. The deep learning architectures, including LSTM, RNN and GRU are trained and evaluated for their effectiveness in distinguishing between legitimate and malicious domain names. The performance of each model is evaluated using standard metrics such as precision, recall, and F1-score. The findings of this research have significant implications for cyber security defense strategies. Our experimental findings illustrate that the proposed model outperforms current state-of-the-art methods in both DGA domain name classification and detection. Our proposed model achieved 99% accuracy for DGA classification. By integrating additional feature extraction and knowledge-based methods our proposed model surpasses existing models. The experimental outcomes suggest that our proposed model gated recurrent unit can achieve 99% accuracy, a 94% recall rate, and a 98% F1-score for the detection and classification of DGA-generated domain names.

A botnet is a network of remote-controlled devices that are infected with malware controlled by botmasters in order to launch cyber attacks. To evade detection, the botmaster frequently changes the domain name of his Command and Control (C&C) server. Notice that most of these types of domain names are generated by domain generation algorithms (DGAs). In this paper, we propose Khaos, a novel DGA with high anti-detection ability based on neural language models and the Wasserstein Generative Adversarial Network (WGAN). The key insight of our research is that real domain names are composed of readable syllables and acronyms, and thus we can arrange syllables and acronyms using neural language models to mimic real domain names. In Khaos, we first find the most common n-grams in real domain names, then tokenize these domain names into n-grams, and finally synthesize new domain names after learning arrangements of n-grams from real domain names. We carry out experiments using a variety of state-of-the-art DGA detection approaches: the statistics-based, the distribution-based, the LSTM-based and the graph-based detection approach. Our experimental results show that the average distance for detecting Khaos under the distribution-based detection approach is 0.64, the AUCs of Khaos under the statistics-based and the LSTM-based detection approach are 0.76 and 0.57, respectively, and the precision of Khaos under the graph-based detection approach is 0.68. Our work proves that the existing detection approaches have big troubles in detecting Khaos, and Khaos has better anti-detection ability than state-of-the-art DGAs. In addition, we find that training the existing detection approach on a dataset including the domain names generated by Khaos can improve its detection ability.

In recent years, domain generation algorithms (DGAs) are the foundational mechanisms for many malware families. Mainly, due to the fact that DGA can generate immense number of pseudo random domain names to associate to a command and control (C2) infrastructures. This paper focuses on to detect and classify the pseudo random domain names without relying on the feature engineering or any other linguistic, contextual or semantics and statistical information by adopting deep learning approaches. A deep learning approach is a complex model of traditional machine learning mechanism that has received renewed interest by solving the long-standing tasks in artificial intelligence (AI) related to the field of natural language processing, image recognition, speech processing and many others. They have immense capability to extract optimal feature representations by taking input as in the form of raw input texts. To leverage this and to transfer the performance enhancement in aforementioned areas towards characterize, detect and classify the DGA generated domain names to a specific malware family, this paper adopts deep learning mechanisms with a known one million benign domain names from Alexa, OpenDNS and a corpus of malicious domain names generated from 17 DGA malware families in real time for training in character and bigram level and a trained model has been evaluated on the OSNIT data set in real-time. Specifically, to understand the effectiveness of various deep learning mechanisms, we used recurrent neural network (RNN), identity-recurrent neural network (I-RNN), long short-term memory (LSTM), convolution neural network (CNN), and convolutional neural network-long short-term memory (CNN-LSTM) architectures. Additionally, to find out an optimal architecture, experiments are done with various configurations of network parameters and network structures. All experiments run up to 1000 epochs with a learning rate set in the range [0.01-0.5]. Overall, deep learning approaches, particularly family of recurrent neural network and a hybrid network (where the first layer is CNN and a subsequent layer is LSTM) have showed significant performance with a highest detection rate 0.9945 and 0.9879 respectively. The main reason is deep learning approaches have inherent mechanisms to capture hierarchical feature extraction and long range-dependencies in sequence inputs.

Botnets pose a major threat to the information security of organizations and individuals. The bots (malware infected hosts) receive commands and updates from the Command and Control (C&C) servers, and hence, contacting and communicating with these servers is an essential requirement of bots. However, once a malware is identified in the infected host, it is easy to find its C&C server and block it, if the domain names of the servers are hard-coded in the malware. To counter such detection, many malwares families use probabilistic algorithms known as domain generation algorithms (DGAs) to generate domain names for the C&C servers. This makes it difficult to track down the C&C servers of the Botnet even after the malware is identified. In this paper, we propose a probabilistic approach for the identification of domain names which are likely to be generated by a malware using DGA. The proposed solution is based on the hypothesis that human generated domain names are usually inspired by the words from a particular language (say English), whereas DGA generated domain names should contain random sub-strings in it. Results show that the percentage of false negatives in the detection of DGA generated domain names using the proposed method is less than 29% across 30 DGA families considered by us in our experimentation.

PurposeLarge language models (LLMs) like Generative Pre-trained Transformer 4 (GPT-4) can assist in detecting errors in radiology reports, but privacy concerns limit their clinical applicability. This study compares closed-source and privacy-compliant open-source LLMs for detecting common errors in radiology reports.Materials and methodsA total of 120 radiology reports were compiled (30 each from X-ray, ultrasound, CT, and MRI). Subsequently, 397 errors from five categories (typographical, numerical, findings-impression discrepancies, omission/insertion, interpretation) were inserted into 100 of these reports; 20 reports were left unchanged. Two open-source models (Llama 3-70b, Mixtral 8x22b) and two commercial closed-source (GPT-4, GPT-4o) were tasked with error detection using identical prompts. The Kruskall–Wallis test and paired t-test were used for statistical analysis.ResultsOpen-source LLMs required less processing time per radiology report than closed-source LLMs (6 ± 2 s vs. 13 ± 4 s; p < 0.001). Closed-source LLMs achieved higher error detection rates than open-source LLMs (GPT-4o: 88% [348/397; 95% CI: 86, 92], GPT-4: 83% [328/397; 95% CI: 80, 87], Llama 3-70b: 79% [311/397; 95% CI: 76, 83], Mixtral 8x22b: 73% [288/397; 95% CI: 68, 77]; p < 0.001). Numerical errors (88% [67/76; 95% CI: 82, 93]) were detected significantly more often than typographical errors (75% [65/86; 95% CI: 68, 82]; p = 0.02), discrepancies between findings and impression (73% [73/101; 95% CI: 67, 80]; p < 0.01), and interpretation errors (70% [50/71; 95% CI: 62, 78]; p = 0.001).ConclusionOpen-source LLMs demonstrated effective error detection, albeit with comparatively lower accuracy than commercial closed-source models, and have potential for clinical applications when deployed via privacy-compliant local hosting solutions.Key PointsQuestionCan privacy-compliant open-source large language models (LLMs) match the error-detection performance of commercial non-privacy-compliant closed-source models in radiology reports?FindingsClosed-source LLMs achieved slightly higher accuracy in detecting radiology report errors than open-source models, with Llama 3-70b yielding the best results among the open-source models.Clinical relevanceOpen-source LLMs offer a privacy-compliant alternative for automated error detection in radiology reports, improving clinical workflow efficiency while ensuring patient data confidentiality. Further refinement could enhance their accuracy, contributing to better diagnosis and patient care.Graphical

Domain Generation Algorithm (DGA) is an algorithm used by malware to generate a large number of domain names, or DGA domains, on a regular basis. This ancient but ever-active technique is the key weapon on which central structure botnets rely. In recent years, many botnets have adopted the DGA algorithm's domain transformation technique to evade detection and blocking, making it extremely difficult for security personnel to detect DGA domain names. In this article, we use the domain names of Alexa's top one million global websites as a white sample. For the DGA sample data, we take the open data of 360NetLab as the black sample to form the data set of this paper. We use 2-Gram model for feature extraction, and construct the DGA domain detection model based on the LightGBM algorithm. Experimental results show that the accuracy of our model is higher than 98%, and compared with the current common classification models, our model has better performance in both time and space.

The use of large language models (LLMs) has exploded since November 2022, but there is sparse evidence regarding LLM use in health, medical, and research contexts. We aimed to summarise the current uses of and attitudes towards LLMs across our campus’ clinical, research, and teaching sites. We administered a survey about LLM uses and attitudes. We conducted summary quantitative analysis and inductive qualitative analysis of free text responses. In August–September 2023, we circulated the survey amongst all staff and students across our three campus sites (approximately n = 7500), comprising a paediatric academic hospital, research institute, and paediatric university department. We received 281 anonymous survey responses. We asked about participants’ knowledge of LLMs, their current use of LLMs in professional or learning contexts, and perspectives on possible future uses, opportunities, and risks of LLM use. Over 90% of respondents have heard of LLM tools and about two-thirds have used them in their work on our campus. Respondents reported using LLMs for various uses, including generating or editing text and exploring ideas. Many, but not necessarily all, respondents seem aware of the limitations and potential risks of LLMs, including privacy and security risks. Various respondents expressed enthusiasm about the opportunities of LLM use, including increased efficiency. Our findings show LLM tools are already widely used on our campus. Guidelines and governance are needed to keep up with practice. Insights from this survey were used to develop recommendations for the use of LLMs on our campus.

Nowadays, malware uses Algorithmically Generated Domains (AGDs) to establish communication with Command and Control (C&C) servers. Dictionary based Domain Generation Algorithm (DGA) selects words from the frequently changed dictionaries to generate AGDs similar to benign domains, which degrades the accuracy of string based detection method. To combat this, we propose a DGA detection method based on DomainGraph and GCN (Graph Convolutional Network) which detects cross-dictionary AGDs based on the association relation between domains instead of lexical features. Starting from the association relation between domains rather than the lexical features of the domain itself, we can detect the unknown AGDs from a known AGD, regardless of the DGA dictionary they use. The proposed method exploits the fact that string association of benign domains is weak, while AGDs' association is strong. DGGCN composes a domain segmentation method, constructs a graph composed of domains (DomainGraph) based on segmentations and adopts GCN to detect AGDs. We conduct the experiments on public datasets under three settings: detecting AGDs generated by familiar dictionaries, unfamiliar dictionaries and confusing dictionaries. The results reveal that DGGCN can detect cross-dictionary AGDs similar to benign domains more accurately and robustly.

Large language models (LLMs) continue to exhibit noteworthy capabilities across a spectrum of areas, including emerging proficiencies across the health care continuum. Successful LLM implementation and adoption depend on digital readiness, modern infrastructure, a trained workforce, privacy, and an ethical regulatory landscape. These factors can vary significantly across health care ecosystems, dictating the choice of a particular LLM implementation pathway. This perspective discusses 3 LLM implementation pathways—training from scratch pathway (TSP), fine-tuned pathway (FTP), and out-of-the-box pathway (OBP)—as potential onboarding points for health systems while facilitating equitable adoption. The choice of a particular pathway is governed by needs as well as affordability. Therefore, the risks, benefits, and economics of these pathways across 4 major cloud service providers (Amazon, Microsoft, Google, and Oracle) are presented. While cost comparisons, such as on-demand and spot pricing across the cloud service providers for the 3 pathways, are presented for completeness, the usefulness of managed services and cloud enterprise tools is elucidated. Managed services can complement the traditional workforce and expertise, while enterprise tools, such as federated learning, can overcome sample size challenges when implementing LLMs using health care data. Of the 3 pathways, TSP is expected to be the most resource-intensive regarding infrastructure and workforce while providing maximum customization, enhanced transparency, and performance. Because TSP trains the LLM using enterprise health care data, it is expected to harness the digital signatures of the population served by the health care system with the potential to impact outcomes. The use of pretrained models in FTP is a limitation. It may impact its performance because the training data used in the pretrained model may have hidden bias and may not necessarily be health care–related. However, FTP provides a balance between customization, cost, and performance. While OBP can be rapidly deployed, it provides minimal customization and transparency without guaranteeing long-term availability. OBP may also present challenges in interfacing seamlessly with downstream applications in health care settings with variations in pricing and use over time. Lack of customization in OBP can significantly limit its ability to impact outcomes. Finally, potential applications of LLMs in health care, including conversational artificial intelligence, chatbots, summarization, and machine translation, are highlighted. While the 3 implementation pathways discussed in this perspective have the potential to facilitate equitable adoption and democratization of LLMs, transitions between them may be necessary as the needs of health systems evolve. Understanding the economics and trade-offs of these onboarding pathways can guide their strategic adoption and demonstrate value while impacting health care outcomes favorably.