Network security and anomaly detection with Big-DAMA, a big data analytics framework

The paper proposes an approach to security information processing in order to detect computer attacks and network anomalies based on big data technologies. The main contribution of the work is in the development, implementation and investigation of the proposed combined framework for processing security data using parallel computing environment and measuring the performance of the implemented system for detection of network attacks and anomalies. The research goal is to increase the performance of attack detection (under the given requirements for accuracy of solutions) compared to the traditional IDS application. The implemented approach is built using the open source systems Snort and Spark. The paper discusses the capabilities and performance assessment of parallel data processing in order to detect computer attacks and network anomalies, as well as key principles of working with big data. The presented main results of an experimental performance evaluation of the applied approach confirm its high efficiency for analyzing network traffic and security events.

Enabled by a rich ecosystem of Machine Learning (ML) libraries, programming using learned models , i.e., Software-2.0 , has gained substantial adoption. However, we do not know what challenges developers encounter when they use ML libraries. With this knowledge gap, researchers miss opportunities to contribute to new research directions, tool builders do not invest resources where automation is most needed, library designers cannot make informed decisions when releasing ML library versions, and developers fail to use common practices when using ML libraries. We present the first large-scale quantitative and qualitative empirical study to shed light on how developers in Software-2.0 use ML libraries, and how this evolution affects their code. Particularly, using static analysis we perform a longitudinal study of 3,340 top-rated open-source projects with 46,110 contributors. To further understand the challenges of ML library evolution, we survey 109 developers who introduce and evolve ML libraries. Using this rich dataset we reveal several novel findings. Among others, we found an increasing trend of using ML libraries: The ratio of new Python projects that use ML libraries increased from 2% in 2013 to 50% in 2018. We identify several usage patterns including the following: (i) 36% of the projects use multiple ML libraries to implement various stages of the ML workflows, (ii) developers update ML libraries more often than the traditional libraries , (iii) strict upgrades are the most popular for ML libraries among other update kinds, (iv) ML library updates often result in cascading library updates, and (v) ML libraries are often downgraded (22.04% of cases). We also observed unique challenges when evolving and maintaining Software-2.0 such as (i) binary incompatibility of trained ML models and (ii) benchmarking ML models. Finally, we present actionable implications of our findings for researchers, tool builders, developers, educators, library vendors, and hardware vendors.

Background: Acute coronary syndrome (ACS) remains one of the leading causes of death globally. Accurate and reliable mortality risk prediction of ACS patients is essential for developing targeted treatment strategies. Existing traditional models offer moderate discriminative value, and do not incorporate contemporary predictors of ACS prognosis. Machine learning (ML) models have emerged as an alternate method that may offer improved risk assessment. This study aims to compare machine learning models with traditional risk scores for predicting all-cause mortality in patients with ACS. Methods: PubMed, EMBASE, Web of Science and Cochrane databases were searched until 1st November 2023 for studies comparing ML models with traditional statistical methods for event prediction of ACS patients. The primary outcome was comparative discrimination measured by C-statistics with 95% confidence intervals between ML models and traditional methods in estimating the risk of all-cause mortality. Results: Ten studies were included (239627 patients). The summary C-statistic of all ML models across all endpoints was 0.89 (95% CI, 0.86-0.92), compared to traditional methods 0.82 (95% CI, 0.79-0.85). The difference in C-statistic between all ML models and traditional methods was 0.07 (p<0.05). Three models undertook external validation, and calibration was inconsistently reported. Conclusion: ML models demonstrated superior discrimination of all-cause mortality for ACS patients compared to traditional risk scores. Despite outperforming well-established prognostic tools such as the GRACE and TIMI scores, current clinical applications of ML approaches remain uncertain, particularly in view of the methodological limitations of existing studies and the need for greater model validation.

Using AI to Predict Patient's Atypical Prescription Based on Individual's Feature-Set

Integrating machine learning (ML) models into healthcare systems is a rapidly evolving field with the potential to revolutionize care delivery. This study aimed to classify fertility rates and identify significant predictors using ML models among reproductive women in Ethiopia. This study utilized eight ML models in 5864 reproductive-age women using Ethiopian Demographic Health Survey (EDHS), 2019 data. Phyton programming language was used to develop these models. Predictors of fertility rate were determined using the feature important techniques. The performance of models was evaluated using accuracy, area under the curve (AUC), precision, recall, F1-score, specificity, and sensitivity. The mean age of participants was 32.7 (± 5.6) years. The random forest classifier (accuracy = 0.901 and AUC = 0.961) followed by a one-dimensional convolutional neural network (accuracy = 0.899 and AUC = 0.958), logistic regression (accuracy = 0.874 and AUC = 0.937), and gradient boost classifier (accuracy = 0.851 and AUC 0.927) were the top performing ML models. Family size, age, occupation, and education with an average importance score of 0.198, 0.151, 0.118, and 0.081, respectively were the top significant predictors of the fertility rate. The best ML models to classify and predict fertility rates were random forest, one-dimensional convolutional neural network, logistic regression, and gradient boost classifier. The findings on important factors of fertility rate can inform targeted public health, programs that address disparities related to family size, occupation, education, and other socioeconomic factors.

The use of machine learning (ML) models in forensic anthropology (FA) has increased in the last half decade; however, there is a lack of a standardized protocol on how to curate, use, and assess ML models. We introduce PUMAA (A Protocol for Utilizing Machine Learning in Forensic Anthropological Analyses), which includes a flowchart and a checklist that forensic practitioners can follow when creating, using, or assessing ML models for forensic research. The five factors that must be assessed in determining an ML model's true performance are explored. The most common types of supervised ML models are also explained in lay terms and accompanied by visuals to increase accessibility of these complex concepts. Although not exhaustive, examples of information that should be reported have been discussed for seven types of ML models. In addition, their various strengths and limitations are evaluated in order to equip researchers with the necessary tools to make decisions regarding when and how to use ML models. This protocol provides an initial standard for the use of ML in FA.

In the detection process of server network storm attack, the traditional method takes the signal detection algorithm, because the attack number is large, the detection error is big.According to the problem, an improved server network storm attack detection method is proposed based on improved density clustering algorithm, the server network storm attack detection problem is transformed into a multi class classification problem, wrapper features selection model is taken, IDBC network connection record distance calculation method is combined, based on DBSCAN clustering results, the data clustering and attack detection is obtained.Simulation results show that, the improved density clustering algorithm is applied in detection of server network storm attack, it can reduce the detection error, the performance of the detection system is improved.

Machine learning (ML) often provides applicable high-performance models to facilitate decision-makers in various fields. However, this high performance is achieved at the expense of the interpretability of these models, which has been criticized by practitioners and has become a significant hindrance in their application. Therefore, in highly sensitive decisions, black boxes of ML models are not recommended. We proposed a novel methodology that uses complex supervised ML models and transforms them into simple, interpretable, transparent statistical models. This methodology is like stacking ensemble ML in which the best ML models are used as a base learner to compute relative feature weights. The index of these weights is further used as a single covariate in the simple logistic regression model to estimate the likelihood of an event. We tested this methodology on the primary dataset related to cardiovascular diseases (CVDs), the leading cause of mortalities in recent times. Therefore, early risk assessment is an important dimension that can potentially reduce the burden of CVDs and their related mortality through accurate but interpretable risk prediction models. We developed an artificial neural network and support vector machines based on ML models and transformed them into a simple statistical model and heart risk scores. These simplified models were found transparent, reliable, valid, interpretable, and approximate in predictions. The findings of this study suggest that complex supervised ML models can be efficiently transformed into simple statistical models that can also be validated.

We conduct an in-depth investigation of the structural, electronic, vibrational, thermodynamic, and thermoelectric characteristics of Na–Bi-based compounds, specifically tetragonal NaBi, hexagonal NaBi3, and cubic Na3Bi, using advanced first-principles calculations in conjunction with machine learning (ML) models. We used density functional theory (DFT) with spin–orbit coupling (SOC) to figure out the electronic structure, phonon dispersions, and thermoelectric transport using Boltzmann transport theory. Our findings validate the Dirac semimetal nature of cubic Na3Bi and demonstrate varied topological and thermodynamic properties within the Na–Bi family. To speed up the prediction of the thermoelectric figure of merit (ZT) while enhancing interpretability to understand at the feature level, we trained supervised ML models [Random Forest (RF) and Neural Network (NN)] on thermoelectric results from DFT. It is possible to directly compute the figure of merit (ZT) from DFT-derived transport coefficients such as the Seebeck coefficient, electrical conductivity, and thermal conductivity. However, machine learning (ML) models serve as powerful surrogate predictors, enabling rapid screening of derivative compounds and quantitative assessment of feature importance through SHAP (SHapley Additive exPlanations) analysis. At low temperatures, RF models consistently outperformed NN models, but both performed well at high temperatures. SHAP analysis showed that the Seebeck coefficient has the biggest effect on ZT in all regimes. This integrated, physics-informed, and data-driven methodology demonstrates that machine learning can significantly augment first-principles approaches. It accelerates predictions, guides feature prioritisation, and enhances design capabilities. The developed workflow provides a generalizable and interpretable framework for the predictive modeling of advanced topological thermoelectric materials.

SummarySustained engagement in HIV care and adherence to antiretroviral therapy (ART) are essential for achieving the UNAIDS “95-95-95” targets. Despite increased ART access, disengagement from care remains a significant issue, particularly in sub-Saharan Africa. Traditional machine learning (ML) models have shown moderate success in predicting care disengagement, which would enable early intervention. We develop an enhanced large language model (LLM) fine-tuned with electronic medical records (EMRs) to predict people at risk of disengaging from HIV care in Tanzania and to provide interpretative insights into modifiable risk factors.MethodsWe developed a novel AI model by enhancing a pre-trained LLM (LLaMA 3.1, an open-source pre-trained LLM released by Meta) using routinely collected EMRs from Tanzania’s National HIV Care and Treatment Program from January 1, 2018, to June 30, 2023 (4,809,765 records for 261,192 people) to identify people at risk of disengaging from HIV care or developing adverse outcomes. Outcomes included risk of ART non-adherence, non-suppressed viral load, and loss to follow-up. Models were evaluated internally (Kagera region) and externally (Geita region), with performance compared against state-of-art ML models and zero-shot LLMs. Additionally, a team of HIV physicians in Tanzania assessed the LLM’s predictions along with LLM provided justifications for a subset of patient records to evaluate their clinical relevance and reasoning.FindingsThe enhanced LLMs consistently outperformed the supervised ML model and zero-shot LLMs across all outcomes in both internal and external validation datasets. When focusing on the 25% of PLHIV predicted as most likely to lost-to-follow-up (LTFU), the model correctly identified 78% (2,515 of 3,224) of people living with HIV (PLHIV) genuinely at risk in internal validation and 73% (7,105 of 9,733) in external validation. Attention score analysis indicated that the enhanced LLM focused on keywords such as gaps in follow-up care and ART adherence. The human expert evaluation showed alignment between clinician assessments and the LLM’s predictions in 65% of cases, with experts finding the model’s justifications reasonable and clinically relevant in 92.3% of aligned cases.InterpretationIf implemented in HIV clinics, this LLM-based AI model could help allocate resources efficiently and deliver targeted interventions, improving retention in care and advancing the UNAIDS “95-95-95” targets. By functioning like a clinician–analyzing patient summaries, predicting risks, and offering reasoning–the enhanced LLM could be integrated into clinical workflows to complement human expertise, facilitating timely interventions and informed decision-making. If implemented widely, this human-AI collaboration has the potential to improve health outcomes for people living with HIV and reduce onward transmission.FundingThe study was supported by a grant from the US National Institutes of Health (NIH): NIH NIMH 1R01MH125746.

The future 6G network will be able to connect massive and various unmanned vehicles (UxVs) into the network, bringing novel requirements toward UxV network security and data privacy. Deploying machine learning (ML)-based intrusion detection on UxVs can be one promising approach. The conventional approach cannot fulfill security and privacy requirements, as the structure can be more flexible, and there may be a lack of a central node in the 6G-based UxV network. Moreover, UxV clients can exhibit high heterogeneity with extremely class-imbalanced training samples. With the presence of federated learning (FL), UxVs can collaborate to train and update ML models while preserving data privacy in the UxV security critical scenario. However, most past works focused on centralized approaches, using cloud servers to assist intrusion detection and service deployments. This article introduces an FL framework with a decentralized design for training ML models on UxVs using intrusion detection as a study case. This scheme can be more flexible, as UxV clients can collaborate to train and synchronize models without the central server. Simulation experiments were conducted to evaluate the efficiency of the proposed approach. Experimental results and theoretical analysis confirm that the proposed method outperforms baseline models with FedAvg and local-trained client models.

BackgroundPhysical activity (PA) level is associated with multiple health benefits during early childhood. However, inconsistency in the methods for quantification of PA levels among preschoolers remains a problem.ObjectiveThis study aimed to develop PA intensity cut points for wrist-worn accelerometers by using machine learning (ML) approaches to assess PA in preschoolers.MethodsWrist- and hip-derived acceleration data were collected simultaneously from 34 preschoolers on 3 consecutive preschool days. Two supervised ML models, receiver operating characteristic curve (ROC) and ordinal logistic regression (OLR), and one unsupervised ML model, k-means cluster analysis, were applied to establish wrist-worn accelerometer vector magnitude (VM) cut points to classify accelerometer counts into sedentary behavior, light PA (LPA), moderate PA (MPA), and vigorous PA (VPA). Physical activity intensity levels identified by hip-worn accelerometer VM cut points were used as reference to train the supervised ML models. Vector magnitude counts were classified by intensity based on three newly established wrist methods and the hip reference to examine classification accuracy. Daily estimates of PA were compared to the hip-reference criterion.ResultsIn total, 3600 epochs with matched hip- and wrist-worn accelerometer VM counts were analyzed. All ML approaches performed differently on developing PA intensity cut points for wrist-worn accelerometers. Among the three ML models, k-means cluster analysis derived the following cut points: ≤2556 counts per minute (cpm) for sedentary behavior, 2557-7064 cpm for LPA, 7065-14532 cpm for MPA, and ≥14533 cpm for VPA; in addition, k-means cluster analysis had the highest classification accuracy, with more than 70% of the total epochs being classified into the correct PA categories, as examined by the hip reference. Additionally, k-means cut points exhibited the most accurate estimates on sedentary behavior, LPA, and VPA as the hip reference. None of the three wrist methods were able to accurately assess MPA.ConclusionsThis study demonstrates the potential of ML approaches in establishing cut points for wrist-worn accelerometers to assess PA in preschoolers. However, the findings from this study warrant additional validation studies.

BackgroundAdvances in automated data processing and machine learning (ML) models, together with the unprecedented growth in the number of social media users who publicly share and discuss health-related information, have made public health surveillance (PHS) one of the long-lasting social media applications. However, the existing PHS systems feeding on social media data have not been widely deployed in national surveillance systems, which appears to stem from the lack of practitioners and the public’s trust in social media data. More robust and reliable data sets over which supervised ML models can be trained and tested reliably is a significant step toward overcoming this hurdle. The health implications of daily behaviors (physical activity, sedentary behavior, and sleep [PASS]), as an evergreen topic in PHS, are widely studied through traditional data sources such as surveillance surveys and administrative databases, which are often several months out-of-date by the time they are used, costly to collect, and thus limited in quantity and coverage.ObjectiveThe main objective of this study is to present a large-scale, multicountry, longitudinal, and fully labeled data set to enable and support digital PASS surveillance research in PHS. To support high-quality surveillance research using our data set, we have conducted further analysis on the data set to supplement it with additional PHS-related metadata.MethodsWe collected the data of this study from Twitter using the Twitter livestream application programming interface between November 28, 2018, and June 19, 2020. To obtain PASS-related tweets for manual annotation, we iteratively used regular expressions, unsupervised natural language processing, domain-specific ontologies, and linguistic analysis. We used Amazon Mechanical Turk to label the collected data to self-reported PASS categories and implemented a quality control pipeline to monitor and manage the validity of crowd-generated labels. Moreover, we used ML, latent semantic analysis, linguistic analysis, and label inference analysis to validate the different components of the data set.ResultsLPHEADA (Labelled Digital Public Health Dataset) contains 366,405 crowd-generated labels (3 labels per tweet) for 122,135 PASS-related tweets that originated in Australia, Canada, the United Kingdom, or the United States, labeled by 708 unique annotators on Amazon Mechanical Turk. In addition to crowd-generated labels, LPHEADA provides details about the three critical components of any PHS system: place, time, and demographics (ie, gender and age range) associated with each tweet.ConclusionsPublicly available data sets for digital PASS surveillance are usually isolated and only provide labels for small subsets of the data. We believe that the novelty and comprehensiveness of the data set provided in this study will help develop, evaluate, and deploy digital PASS surveillance systems. LPHEADA will be an invaluable resource for both public health researchers and practitioners.

With the rapid development of Internet technology, computer has become an indispensable part of modern society. However, with the popularization of computer technology, the problem of network security is becoming increasingly prominent. In recent years, China's network security incidents occur frequently. As a means of attack, computer deception has seriously threatened national information security, social stability and people's life. Therefore, it is of great practical significance to study the role of computer deception in network security. Its purpose is to mislead and deceive users or systems by various means to achieve the purpose of illegally obtaining information, damaging the stability of the system or carrying out other illegal activities. Based on this, this paper aims to explore the role of computer deception in network security. Firstly, the concept, content and present situation of computer network security are expounded, and then the necessity of computer network security management and protection is analyzed. Secondly, the related technology of computer deception is introduced. Finally, the application of computer spoofing in network security is studied, and the effectiveness of computer spoofing in preventing and detecting network attacks is revealed. In addition, the ethical issues of computer deception and its potential in future cybersecurity strategies are discussed. Research shows that computer deception can not only enhance the defense mechanism, but also improve the organization's cognition and response ability to network threats, and become an indispensable security strategy. By evaluating current technologies and methods, this paper provides practical recommendations for researchers and practitioners to effectively use computer spoofing techniques to improve network security.

ObjectivesThis study aimed to develop and validate machine learning (ML) models to predict all-cause hospital admissions and 90-day readmissions using structured, patient-reported survey data.MethodsA cross-sectional survey was conducted between 3 July 2021 and 18 December 2022, among US adults aged ≥18 years with at least one cardiovascular risk factor. Participants were recruited through social media, community pharmacies and outpatient clinics. The final sample included 1318 participants. Primary outcomes were any all-cause hospitalisation and readmission within 90 days. Eight supervised ML models were trained using an 80:20 train–test split and 10-fold cross-validation. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC), precision, recall, F1 score and calibration metrics. SHapley Additive exPlanations (SHAP) values identified key predictors.ResultsAmong 1318 participants, 35.0% reported at least one hospitalisation and 10.4% reported a 90-day readmission. The Extra Trees (ET) model demonstrated the best performance across both outcomes. For hospitalisation, ET achieved an AUROC of 0.93, precision of 0.83 and recall of 0.87. For readmission, AUROC was 0.99 with precision of 0.95 and recall of 0.96. SHAP analysis identified heart disease, medication burden, race/ethnicity, employment and insurance status as the most influential predictors.DiscussionPatient-reported data reflecting behavioural, social and clinical factors can predict hospitalisations with high accuracy, complementing traditional EHR-based models.ConclusionsIntegrating such patient-reported and behavioural data into electronic health records could enable earlier identification of high-risk individuals and support targeted, preventive interventions to improve healthcare outcomes.