Scalable Solutions for Automated Single Pulse Identification and Classification in Radio Astronomy

Searching for pulsar signals in radio astronomy data sets is a difficult task. The data sets are extremely large, approaching the petabyte scale, and are growing larger as instruments become more advanced. Big Data brings with it big challenges. Processing the data to identify candidate pulsar signals is computationally expensive and must utilize parallelism to be scalable. Labeling benchmarks for supervised classification is costly. To compound the problem, pulsar signals are very rare, e.g., only 0.05% of the instances in one data set represent pulsars. Furthermore, there are many different approaches to candidate classification with no consensus on a best practice. This dissertation is focused on identifying and classifying radio pulsar candidates from single pulse searches. First, to identify and classify Dispersed Pulse Groups (DPGs), we developed a supervised machine learning approach that consists of RAPID (a novel peak identification algorithm), feature extraction, and supervised machine learning classification. We tested six algorithms for classification with four imbalance treatments. Results showed that classifiers with imbalance treatments had higher recall values. Overall, classifiers using multiclass RandomForests combined with Synthetic Majority Oversampling TEchnique (SMOTE) were the most efficient; they identified additional known pulsars not in the benchmark, with less false positives than other classifiers. Second, we developed a parallel single pulse identification method, D-RAPID, and introduced a novel automated multiclass labeling (ALM) technique that we combined with feature selection to improve execution performance. D-RAPID improved execution performance over RAPID by a factor of 5. We also showed that the combination of ALM and feature selection sped up the execution performance of RandomForest by 54% on average with less than a 2% average reduction in classification performance. Finally, we proposed CoDRIFt, a novel classification algorithm that is distributed for scalability and employs semi-supervised learning to leverage unlabeled data to inform classification. We evaluated and compared CoDRIFt to eleven other classifiers. The results showed that CoDRIFt excelled at classifying candidates in imbalanced benchmarks with a majority of non-pulsar signals (>95%). Furthermore, CoDRIFt models created with very limited sets of labeled data (as few as 22 labeled minority class instances) were able to achieve high recall (mean = 0.98). In comparison to the other algorithms trained on similar sets, CoDRIFt outperformed them all, with recall 2.9% higher than the next best classifier and a 35% average improvement over all eleven classifiers. CoDRIFt is customizable for other problem domains with very large, imbalanced data sets, such as

The detection of zero-day vulnerabilities remains one of the most critical challenges in modern cybersecurity. Traditional detection systems, primarily reliant on signature-based methods, are ineffective against unknown or novel attacks. This book chapter explores the integration of adaptive machine learning algorithms for real-time zero-day vulnerability detection, highlighting the transition from conventional approaches to intelligent, dynamic solutions. Emphasis is placed on the evolution of machine learning techniques, including supervised, unsupervised, and semi-supervised learning, which enable the identification of previously unseen threats. The chapter also investigates the challenges faced in deploying machine learning models in real-time environments, such as high-dimensionality data, feature selection, and the need for continuous adaptation to emerging attack patterns. Additionally, it covers key tools and frameworks, such as TensorFlow, Apache Spark, and Apache Kafka, which support the development of scalable, low-latency detection systems. The potential of these frameworks to handle large-scale data streams while maintaining real-time performance is critical for enhancing the resilience of cybersecurity systems. By leveraging machine learning, organizations can significantly improve their capacity to identify and mitigate zero-day vulnerabilities before they cause substantial damage. This chapter provides an in-depth analysis of these techniques, offering insights into their practical applications and their contributions to advancing the field of cybersecurity.

From past ten years, the area of Feature Selection is favorite among researchers because of the increasing size of the datasets. Several existing feature selection algorithms do not scale well with the large datasets. Sophisticated feature selection tools also do not work well with large datasets on low configuration machine. This work implements feature selection using spark cluster computing to have a scalable solution. In the proposed algorithm, large scale dataset is partitioned vertically into several small datasets. Spark distributes it in form of RDD (Resilient Distributed Dataset) among the machines in the cluster. The information gain based filter is applied to RDDs on each machine to evaluate ranks of features on multiple machines in parallel. The sub-results are collected and combined on master machine and attributes with high information gain are selected as relevant attributes. The program is run on different number of cores with different dataset sizes having different number of attributes and samples. Proposed approach shows good scalability and effective reduction in overall execution times with higher number of cores.

Organization that scatters over different regions that perform multi-state business transaction are always interested I identifying novel patterns of interest given augmenting nosiness volumes. On a daily basis, we encounter an unparalleled increase in data from many sources, which adds to the concept of big data in terms of its size, speed, and diversity. These datasets provide significant obstacles to analytics frameworks and processing resources, so making the total study arduous for extracting useful information promptly. Therefore, in order to overcome these types of difficulties, it is crucial to create a very effective framework for analyzing large amounts of data. Therefore, in order to tackle these difficulties by harnessing non-linear connections from extensive and complex information, analytics frameworks are using machine (ML) and (DL) methods. Apache Spark is widely recognized as the most efficient tool for processing large amounts of data. It is particularly useful for solving complex machine learning problems that need several iterations. Spark MLlib, a distributed machine learning library, is used for this purpose. When dealing with research problems in the real world, architectures like Long Short-Term Memory in deep learning are a useful method for addressing practical challenges like as decreased accuracy, long-term sequence dependence, and the problem of disappearing and expanding gradients in traditional deep architectures. This study proposes a very efficient analytics system that combines a progressive machine learning method with Spark-based linear models, Multilayer Perceptron , and Long Short-Term Memory. The accuracy of the system's predictions is improved by the use of a two-stage cascade structure. The architecture that we provide makes it possible to organize statistical analysis of large amounts of data in a way that is not only scalable but also effective. We applied the cascading structure on two different real-world datasets in order to illustrate the effectiveness of our framework. Furthermore, the results of the experiments show that our analytical framework is superior to the approaches that are considered to be state-of-the-art in terms of classification accuracy.

ion called Resilient Distributed Datasets (RDDs) [88]. RDDs is proposed to let programmers perform in-memory computations on large clusters in a fault-tolerant manner. This abstraction is motivated by two types of applications: iterative algorithms and interactive data mining tools that current computing frameworks handle inefficiently. As the cost of implementing machine learning algorithms as low-level MapReduce jobs on varying data and machine cluster sizes can be prohibitive, SystemML [89] is proposed to express machine learning algorithms in a higher-level language which are Scala. http://www.scala-lang.org. 56 compiled and executed in a MapReduce environment. MLbase [90] is a scalable machine learning system for both end-users and machine learning researchers. Two main functions are provided by MLbase: it enables machine learning researchers to implement scalable machine learning methods without requiring deep system knowledge; it provides a novel optimizer to select and dynamically adapt the choice of learning algorithms. Similar to SystemML, MLbase applies simple declarative way to specify machine learning tasks.

Sickle cell disease (SCD) is a common serious genetic disease, which has a severe impact due to red blood cell (RBCs) abnormality. According to the World Health Organisation, 7 million newborn babies each year suffer either from the congenital anomaly or from an inherited disease, primarily from thalassemia and sickle cell disease. In the case of SCD, recent research has shown the beneficial effects of a drug called hydroxyurea/hydroxycarbamide in modifying the disease phenotype. The clinical management of this disease-modifying therapy is difficult and time consuming for clinical staff. This includes finding an optimal classifier that can help to solve the issues with missing values, multi-class datasets, and features selection. For the classification and discriminant analysis of SCD datasets, 7 classifiers based on machine learning models are selected representing linear and non-linear methods. After running these classifiers with a single model, the results revealed that a single classifier has provided us with effective outcomes in terms of the classification performance evaluation metric. In order to produce such an optimal outcome, this research proposed and designed combined classifiers (ensemble classifiers) among the neural network’s models, the random forest classifier, and the K-nearest neighbour classifier. In this aspect, combining the levenberg-marquardt algorithm, the voted perceptron classifier, the radial basis neural classifier, and random forest classifier obtain the highest rate of performance and accuracy. This ensemble classifier receives better results during the training set and testing set process. Recent technology advances based on smart devices have improved the medical facilities and become increasingly popular in association with real-time health monitoring and remote/personal health-care. The web-based system developed under the supervision of the haematology specialist at the Alder Hey Children’s Hospital in order to produce such an effective and useful system for both patients and clinicians. To sum up, the simulation experiment concludes that using machine learning and the web-based system platforms represents an alternative procedure that could assist healthcare professionals, particularly for the specialist nurse and junior doctor to improve the quality of care with sickle cell disorder.

Identification of high-risk genes and classification of acute myocardial infarction patients utilizing deep learning in a restricted cohort.

A novel firefly algorithm approach for efficient feature selection with COVID-19 dataset

Twitter integrates with streaming data technologies and machine learning to add new value to healthcare. This paper presented a real‐time system to predict breast cancer based on streaming patient’s health data from Twitter. The proposed system consists of two major components: developing an offline building model and an online prediction pipeline. For the first component, we made a correlation between the features to determine the correlation between features and reduce the number of features from the Breast Cancer Wisconsin Diagnostic dataset. Two feature selection algorithms are recursive feature elimination and univariate feature selection algorithms which are applied to features after correlation to select the essential features. Four decision trees, logistic regression, support vector machine, and random forest classifier have been used on features after correlation and feature selection. Also, hyperparameter tuning and cross‐validation have been applied with machine learning to optimize models and enhance accuracy. Apache Spark, Apache Kafka, and Twitter Streaming API are used to develop the second component. The best model with the highest accuracy obtained from the first component predicts breast cancer in real time from tweets’ streaming. The results showed that the best model is the random forest classifier which achieved the best accuracy.

<p>A reasonable distributed memory-based Computing system for machine learning is Apache Spark. Spark is being superior in computing when compared with Hadoop. Apache Spark is a quick, simple to use for handling big data that has worked in modules of Machine Learning, streaming SQL, and graph processing. We can apply machine learning algorithms to big data easily, which makes it simple by using Spark and its machine learning library MLlib, even this can be made simpler by using the Python API PySpark. This paper presents the study on how to develop machine learning algorithms in PySpark. </p>

The exponential increase in Medical data and Computer-aided diagnostic tools has made it easier for Machine Learning (ML) Researchers to extract valuable insights from medical images resulting in better patient outcomes during Medical Summarization. According to current statistics, 8.8 million people died every year due to cancer death. It is therefore imperative to use these available image datasets for better diagnosis, increased patient outcomes and recovery from deadly diseases. As a result, this study proposes a Colorectal Cancer classification model for detecting KRAS mutation status of Patients through their Radiomic images using the transfer learning of a deep learning pipeline on Apache Spark. The datasets will be curated from National Cancer Institute (NCI), features extraction or classification of Computed tomography(CT) images into CT- based handcrafted radiomic signatures with a Convolutional Neural Network(CNN) using the Apache Spark framework, the system uploads the segmented scanning images to the High Distributed File System (HDFS), Confidence Interval(CI), Area Under Curve(AUC), Precision and Recall for Validation cohort, Docker for app deployment and containerization of the model developed for reproducibility, transfer learning and model reuse. The result would be a real-time model using Deep learning pipelines with Apache Spark and Keras Tensor flow for KRAS Mutation detection using Colorectal Cancer images. In conclusion, this research project would produce a scalable and reproducible model for the faster diagnosis of Colorectal Cancer through the KRAS mutation status of CRC Patients. Citation Format: Mary Adetutu Adewunmi. Scalable colorectal cancer (CRC) diagnosis model with arterial and unenhanced phases of KRAS mutational status using Apache Spark [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 6391.

A modified genetic algorithm and weighted principal component analysis based feature selection and extraction strategy in agriculture

The sheer boom in data creation from multiple sources beginning from IoT sensors to social media has changed the technology landscape, rendering predictive analytics in big data environments more important than ever before. This work suggests a hybrid machine learning model for improving predictive accuracy, computational performance, and scalability in big data environments. The Model employs Random Forest (RF) for strong feature selection and Gradient Boosted Decision Trees (GBDT) for enhanced classification accuracy, both optimised using Apache Spark's MLlib in a distributed setting. The Model takes advantage of hybridisation to beat the single-algorithm model drawback in accommodating high-dimensional, heterogeneous data. The architecture is designed to handle data in real time and includes a dynamic pre-processing layer, parallel training pipeline, and continuous evaluation modules. Performance metrics like accuracy, precision, recall, F1 score, and AUC are used to validate performance against benchmark data sets. System-level metrics like latency, Throughput, and scalability are also monitored to validate usability for real-world deployments. Experiments using healthcare and e-commerce industry datasets yield better prediction power and operational effectiveness than single ML models. Visualisation using scatter plots and 3D graphs shows sharp jumps in model accuracy and processing time for different volumes of data. It is a scalable, adaptable, and precise predictive analytics tool for the big data era. Ease of use of the solution with Apache Hadoop and Spark deployment is a willingness to be utilised at the enterprise level.

This paper will report on the progress in the research field of radio astronomy in Malaysia and also in South-East Asia. Radio astronomy research can be said to have only started being done actively in Malaysia in 2005. Fundamental setup and some basic research were done in the early parts. Jupiter radio bursts as well as radio meteor counting were the first few projects. Currently a medium-sized radio telescope which was recently purchased is being used to undergo more advanced research such as observing spectrum of galaxy clusters in order to study Dark Matter. This paper will also report on the latest results on the survey of Radio Frequency Interference (RFI), which is currently done on chosen candidate sites in Malaysia in order to setup on it a large radio telescope in the future. The best site with the lowest averaged RFI level is found at Langkawi (−100.33 dBm or 4.4 × 106 Jy). The author is the chair for the Radio Astronomy Working Group for the South-East Asia Astronomy Network (SEAAN) and the Malaysian team is very much an important part of this group. This group will eventually be a research hub for the global projects of the Global-Very Long Baseline Interferometer (Global-VLBI) and the Square Kilometre Array (SKA).

The receiver plays a very important role in the research field of deep space exploration and radio astronomy, and the cryogenic low noise amplifier (LNA) is the key component of their receiving system. The LNA's noise performance is greatly improved when operating at extremely low temperatures. In this paper, we designed a VHF/UHF cryogenic LNA. The LNA adopts the topology of two-stage cascaded amplification. By adopting the structure of series negative feedback and parallel negative feedback and accurate input and output matching, it not only realizes the optimization of input return loss and noise, but also broadens the operating bandwidth and improves stability. Based on this structure, the design of a broadband LNA with high gain, low noise, and better return loss is realized in this paper. This design is tested at 300 K, 77 K, and 15K respectively. The experimental results show that the gain of the VHF/UHF (100-700 MHz) LNA is greater than 32.3 dB, the return loss of the input and output ends are better than 10 dB, and the noise temperature is less than 95.83 K at 300 K, the noise temperature is less than 23.83 K and 17.90 K at the physical temperature of 77 K and 15 K, respectively.