Spark for Social Science

Enterprise IT infrastructures experience a transformation because of rapid cloud computing adoption, especially Amazon Web Services (AWS), which delivers scalability, flexibility, and cost efficiency. AWS users encounter difficulties optimizing their performance because they face problematic situations combining workload variability with complex resource distribution systems and delay issues. The implementation of Artificial Intelligence constructs an evaluation process to boost performance optimization in Amazon Web Services (AWS) cloud systems. AI-powered insights implemented through machine learning algorithms with predictive analytics capabilities and automation functionality improve workload distribution and enhance auto-scaling efficiency while reducing operational expenses. Processor analysis of historical data combined with real-time metrics allows AI algorithms to make proactive decisions about resource administration, detect anomalies, and perform innovative management. The research evaluates performance-altering AI-based approaches that enhance AWS services, including EC2, Lambda, and Sage Maker. Implementing AI-based optimizations generates copious improvements in computational efficiency and cost performance based on experimental data. Thanks to AI, cloud performance management strategies will fundamentally transform since they endorse efficient AWS environments that align with business requirements.

BackgroundWhole-genome sequencing (WGS) plays an increasingly important role in clinical practice and public health. Due to the big data size, WGS data analysis is usually compute-intensive and IO-intensive. Currently it usually takes 30 to 40 h to finish a 50× WGS analysis task, which is far from the ideal speed required by the industry. Furthermore, the high-end infrastructure required by WGS computing is costly in terms of time and money. In this paper, we aim to improve the time efficiency of WGS analysis and minimize the cost by elastic cloud computing.ResultsWe developed a distributed system, GT-WGS, for large-scale WGS analyses utilizing the Amazon Web Services (AWS). Our system won the first prize on the Wind and Cloud challenge held by Genomics and Cloud Technology Alliance conference (GCTA) committee. The system makes full use of the dynamic pricing mechanism of AWS. We evaluate the performance of GT-WGS with a 55× WGS dataset (400GB fastq) provided by the GCTA 2017 competition. In the best case, it only took 18.4 min to finish the analysis and the AWS cost of the whole process is only 16.5 US dollars. The accuracy of GT-WGS is 99.9% consistent with that of the Genome Analysis Toolkit (GATK) best practice. We also evaluated the performance of GT-WGS performance on a real-world dataset provided by the XiangYa hospital, which consists of 5× whole-genome dataset with 500 samples, and on average GT-WGS managed to finish one 5× WGS analysis task in 2.4 min at a cost of $3.6.ConclusionsWGS is already playing an important role in guiding therapeutic intervention. However, its application is limited by the time cost and computing cost. GT-WGS excelled as an efficient and affordable WGS analyses tool to address this problem. The demo video and supplementary materials of GT-WGS can be accessed at https://github.com/Genetalks/wgs_analysis_demo.

Cloud Database Management System (CDBMS) is one of the potential services provided by various Cloud Service Providers. Cloud providers cope with different users, different data and processing or analysis of different data. Traditional Database Management Systems are insufficient to handle such variety of data, users and their requirements. Hence, at the conceptual layer of CDBMS, traditional SQL, Oracle and many more Database Languages are insufficient to provide proper services to their users. HIVE and Pig are the different types of languages which are suitable for the cloud environment which can handle such huge amount of data. In this paper, performance comparison of 3-Node cluster and Cloud Based Cluster provided by the Amazon Web Services is being done. We have compared the processing of structured data with the help of different queries provided by HIVE tool on 3-Node cluster and Amazon Web Service (AWS) cluster. It has been concluded that HIVE queries on AWS cluster gives better results as compared to 3-Node cluster.

Summary Public cloud providers, such as the Microsoft Azure and the Amazon Web Services (AWS), offer a wide variety of virtual machines, with different specifications, and prices, that are enabling users to run high-performance programs without buying specialized and expensive hardware. Moreover, some providers, such as AWS, allow the user to bid for lower cost virtual machines, called Spot Instances. Nonetheless, these machines may be terminated within a few minutes of warning at the provider discretion. In this work, we leveraged the SPITS programming model to implement a high-performance and fault-tolerant seismic processing application that is proper for execution on Spot instances and analyzed how different virtual machines from the AWS may affect the performance and the price of the computation. Our experimental results indicate that Spot instances have similar performance to regular instances but are roughly three times less expensive. Finally, we show that AWS groups virtual machines in Availability Zones and that selecting virtual machines from different zones may also affect the total execution cost.

Large language models (LLMs) are ubiquitously powerful but prohibitively expensive to train, often requiring thousands of compute devices, typically GPUs. To reduce the cost of training LLMs for customers, Amazon Web Services (AWS) launched the Amazon EC2 trn1 instances, powered by AWS Trainium, an Amazon's homegrown deep learning accelerator, as an alternative to distributed LLM training. The trn1 instances provide a high-performance LLM training solution at a lower cost compared to their GPU-based counterpart, the p4d instances, which are powered by Nvidia A100 GPUs. This paper describes the design and development of the Neuron Distributed Training Library, a component of the AWS Neuron SDK, which enables distributed training of large language models on AWS Trainium. Neuron Distributed Training Library supports a variety of existing distributed training techniques with unified interfaces, and provides features to address trn1-specific challenges as well. Our evaluation shows that trn1 instances, specifically the trn1.32xlarge, achieve better or comparable performance (up to 24.6% improvement) while offering significant lower costs (up to 46.3% cost saving) in selected workloads when compared to p4d.24xlarge instances. As a result, AWS Trainium has been adopted for training numerous external and internal models, showcasing its high-performance and cost-effectiveness. Several supported open-source LLMs are accessible via HuggingFace Optimum Neuron.

One of the major difficulties for many laboratories setting up proteomics programs has been obtaining and maintaining the computational infrastructure required for the analysis of the large flow of proteomics data. We describe a system that combines distributed cloud computing and open source software to allow laboratories to set up scalable virtual proteomics analysis clusters without the investment in computational hardware or software licensing fees. Additionally, the pricing structure of distributed computing providers, such as Amazon Web Services, allows laboratories or even individuals to have large-scale computational resources at their disposal at a very low cost per run. We provide detailed step-by-step instructions on how to implement the virtual proteomics analysis clusters as well as a list of current available preconfigured Amazon machine images containing the OMSSA and X!Tandem search algorithms and sequence databases on the Medical College of Wisconsin Proteomics Center Web site ( http://proteomics.mcw.edu/vipdac ).

Abstract—In this paper, we use different deep learning techniques for the classification and detection of Escherichia coli (E. coli) through DNA analysis, deployed on Amazon Web Services to leverage cloud computing capabilities. E. coli is a common bacterium found in the intestines of humans and animals; while many strains are harmless, pathogenic variants can cause severe foodborne illnesses and outbreaks. Therefore, rapid and accurate detection is crucial for ensuring food safety and public health. Utilizing a convolutional neural network architecture, we analyze genomic sequences to accurately differentiate harmful E. coli strains from non-pathogenic ones. The model is trained on a diverse dataset of E. coli genomic data, achieving high accuracy in classification. By deploying this solution on AWS, we enable real-time processing and scalability, enhancing the speed and reliability of microbial detection. This innovative approach contributes to improving food safety and public health through efficient pathogen identification. Keywords— Escherichia coli (E. coli), Deep Learning, DNA Classification, Genomic Data Analysis, Pathogen Detection, Food Safety, Public Health, Machine Learning Algorithms, Cloud Computing, Amazon Web Services (AWS), Bioinformatics, Microbial Identification, Data Scalability, Real-time Processing

Mainframe systems have a track record of reliability, but they need help in today’s fast-changing business world. These include high upkeep expenses and limited room for growth. This paper looks at three top cloud platforms—Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP)—to update old mainframe programs. We compare them in detail, looking at what each does best. AWS has a wide range of tools to move data. Azure works well with cloud and on-site systems and fits into company setups. GCP leads in AI and machine learning. Azure is the top option for updating mainframes, offering compatibility with various cloud configurations, integration with current company infrastructure, and sophisticated data analysis driven by AI. These conclusions provide valuable information for companies transitioning mainframe systems to the cloud, enabling expansion, reduced expenses, and future-ready IT systems.

Cloud computing has revolutionized analysis of large datasets. This tutorial provides a comprehensive, practical guide for research groups seeking to leverage cloud platforms for data analysis. The tutorial covers the foundations of cloud computing, including its history, rationale, and use cases for research, followed by detailed comparisons of the 3 major platforms: Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure. Side-by-side comparisons of services, costs, ease of use, and selection guidelines assist researchers in choosing the most appropriate platform. A complete step-by-step example using hospital price transparency data demonstrates the entire workflow from account creation through results retrieval, enabling researchers to begin productive cloud-based analysis within hours.

There is an ever increasing need of storage capacity for storage of digital archives and historical datadigital preservation, because of regulatory and compliance requirements. There is an increasing interest in disk based archival system. Major technical challenges in creating large disk based storage archive are - providing large capacity at low costs, large read and write throughput, data integrity and sustaining hardware and operating system refresh. In this paper we present the architecture and working principle of an archival storage system that uses an erasure-coded redundancy scheme. We present the design of a Quality of Service (QoS) framework that tries to achieve an optimum balance between file availability, performance and system availability. The design includes a file encoding and placement scheme that allows files to be read from the archive without the need to access any metadata. Finally, we present the results obtained from running an experimental setup on Amazon Web Services.

Cybersecurity training activities require specialized environments, typically called cyber ranges, to make it possible for trainees to acquire not only security knowledge, but also practical security skills. However, the setup of these training environments is a tedious task, which hinders the wider use of cyber ranges for security training. In its turn, this has a negative impact on the development of the cybersecurity workforce that is exceedingly necessary in our network-centric society. In this paper we introduce our approach of using the Amazon Web Services (AWS) Elastic Compute Cloud (EC2) public cloud for cyber range deployment, thus making it possible to conduct cybersecurity training activities at scale and at a relatively low cost. Our system was implemented by extending the functionality of the cyber range instantiation system CyRIS that is available as open source on GitHub. We evaluated our implementation from several perspectives, demonstrating that public cloud deployment can provide similar functionality and performance compared to local server or private cloud deployment, while avoiding the high purchase and management costs associated to those.

Smart Parking Sensor Network project aims to develop a low cost sensor based parking system to map the usage of parking areas. This system consists of sensor nodes which can detect the occupancy of parking space; relay nodes to communicate between sensor nodes and the server; server application to get data from the relay nodes and send data to mobile application; and a mobile application to display the parking areas and the occupancy of the parking areas on a map. The vehicle detection sensor node was developed with low cost and low power sensors and components. The vehicle detection sensor node was designed with magnetic sensor and a distance sensor. The magnetic sensor detects the presence of the vehicle and the distance sensor clarifies it. The mobile application was developed using Android and the server application is hosted in AWS (Amazon Web Services). The tests performed using our system shows promising results, which are presented in this paper.

The Industrial applications based on the Wireless Sensor Networks demands the most efficient techniques for monitoring, controlling and alerting mechanisms with low cost, less power and memory consumption. The dynamic Run-time Reconfigurations in the applications will cater the needs by adding new functionality or replacing an existing functionality with a new one, without having to physically reach the individual node. The component based embedded systems demonstrates the technique of remote monitoring and runtime reconfigurations in an industrial processes. The model is designed with the Renesas (R5F100LE) board. The RL78 controller monitors the threshold values of temperature, voltage and over load conditions. The data retrieval and storage is done through the Amazon Web Server (AWS cloud).

BackgroundUntil recently, sequencing has primarily been carried out in large genome centers which have invested heavily in developing the computational infrastructure that enables genomic sequence analysis. The recent advancements in next generation sequencing (NGS) have led to a wide dissemination of sequencing technologies and data, to highly diverse research groups. It is expected that clinical sequencing will become part of diagnostic routines shortly. However, limited accessibility to computational infrastructure and high quality bioinformatic tools, and the demand for personnel skilled in data analysis and interpretation remains a serious bottleneck. To this end, the cloud computing and Software-as-a-Service (SaaS) technologies can help address these issues.ResultsWe successfully enabled the Atlas2 Cloud pipeline for personal genome analysis on two different cloud service platforms: a community cloud via the Genboree Workbench, and a commercial cloud via the Amazon Web Services using Software-as-a-Service model. We report a case study of personal genome analysis using our Atlas2 Genboree pipeline. We also outline a detailed cost structure for running Atlas2 Amazon on whole exome capture data, providing cost projections in terms of storage, compute and I/O when running Atlas2 Amazon on a large data set.ConclusionsWe find that providing a web interface and an optimized pipeline clearly facilitates usage of cloud computing for personal genome analysis, but for it to be routinely used for large scale projects there needs to be a paradigm shift in the way we develop tools, in standard operating procedures, and in funding mechanisms.

There are many advantages for deploying a mass spectrometry workflow to the cloud. While "cloud computing" can have many meanings, in this case, I am simply referring to a virtual computer that is remotely accessible over the Internet. This "computer" can have as many or few resources (CPU, RAM, disk space, etc.) as your demands require and those resources can be changed as you need without requiring complete reinstalls. Systems can be easily "checkpointed" and restored. I will describe how to deploy virtualized, remotely accessible computers on which you can perform your basic mass spectrometry data analysis. This use is a quite restricted microcosm of what is available under the umbrella of "cloud computing" but it is also the (useful!) niche use for which straightforward how-to documentation is lacking.This chapter is intended for people with little or no experience in creating cloud computing instances. Executing the steps in this chapter, will empower you to instantiate a computer with the performance of your choosing with preconfigured software already installed using the Amazon Web Service (AWS) suite of tools. You can use this for use cases that span when you need limited access to high end computing thru when you give your collaborators access to preconfigured computers to look at their data.