Autonomous data engineering: Reinforcement learning-driven metadata management in cloud-native data ecosystems

Cloud computing is one of the paradigms that have undertaken to deliver the utility computing concept. It views computing as a utility similar to water and electricity. We aim in this paper to make an investigation of two highly efficacious Cloud platforms: Microsoft Azure (Azure) and Amazon Web Services (AWS) from users’ perspectives the point of view of users. We highlight and compare in depth the features of Azure and AWS from users’ perspectives. The features which we shall focus on include (1) Pricing, (2) Availability, (3) Confidentiality, (4) Secrecy, (5) Tier Account and (6) Service Level Agreement (SLA). The study shows that Azure is more appropriate when considering Pricing and Availability (Error Rate) while AWS is more appropriate when considering Tier account. Our user survey study and its statistical analysis agreed with the arguments made for each of the six comparisons factors.

Real time analysis has become a norm in today's fast and competitive business environment. Availability of recent data is a requirement to perform real time analysis. The data warehouse is meant to store historical data for analysis and these are populated from time to time through a process called as extract transform and load (ETL). The ETL process was done during nights or weekends to make the transactional data available at the data warehouse for analysis. But the delay of a day or weekend is not acceptable for real time analysis hence a concept of near real time ETL is developed to perform real time analysis. We explore in this study methods of how near real time ETL could be achieved. We focus on three ways of how near real time ETL could be performed. First is the Meta data management which if done effectively will greatly reduce the development time. Second is the concept of change data capture when implemented with some intelligent would greatly help in making the correct data available for analysis at the warehouse. The third is the use of parallel techniques to partition the flow of ETL and reduce the time window with which the data are available at the warehouse.

Extract, Transform, Load (ETL) processes form the backbone of data manage- ment and consolidation in today’s data-driven enterprises with prevalent large- scale data pipelines. One of the widely used ETL tools is Microsoft SQL Server Integration Services (SSIS), yet its optimization for performance for large-scale data loads remains a challenge. As the volumes of data grow exponentially, inefficient ETL processes create bottlenecks, increased processing time, and ex- haustion of system resources. This work discusses major SSIS optimizations that minimize ETL processing time, allowing for effective and scalable data integration. One of the key areas of optimization is data flow optimization, such as lever- aging the use of the Fast Load mode in OLE DB Destination to perform batch inserts instead of row-by-row. Similarly, Bulk Insert operations can signifi- cantly reduce data movement time. Additionally, buffer size and DefaultBuffer- MaxRows tuning allows SSIS to process data in memory more efficiently, thereby minimizing disk I/O operations. Another major area of focus is source query optimization. With the utiliza- tion of indexed views, partitioned tables, and filtering in the WHERE clause, unnecessary data extraction is avoided, restricting the load on the source sys- tem. NOLOCK hints also minimize database contention in high-concurrency environments. Parallel execution of multiple operations within SSIS can also accelerate execution, with multithreading and batch processing enabling con- current data conversion. Lookup transformations, a common performance bottleneck, can be opti- mized using cache mode, where reference data is pre-loaded instead of querying the database for each row. Furthermore, replacing row-based transformations with set-based operations significantly reduces processing overhead. For incremental data loading, change tracking or CDC (Change Data Cap- ture) enables altered record processing in place of full set loads. This saves time in processing and optimizes utilization of resources. ETL logging and error- handling mechanisms play an important role as well; selective SSIS logging and event-based error-handling mechanisms can prevent performance degradation due to overlogging. Lastly, SSIS package configurations can be tuned by having proper indexing of destination tables, turning off unnecessary constraints during loading, and applying table partitioning to maximize parallel loads of data. By utilizing these SSIS optimizations, organizations can reduce ETL pro- cessing by significant quantities, optimize data pipelines, and overall enhance enterprise-level data integration performance. These approaches make large- scale big-data scale data pipelines have very low latency, thus making SSIS a more efficient and scalable solution for enterprise-level data workflows.

Users of distributed datastores that employ quorum-based replication are burdened with the choice of a suitable client-centric consistency setting for each storage operation. The above matching choice is difficult to reason about as it requires deliberating about the tradeoff between the latency and staleness, i.e., how stale (old) the result is. The latency and staleness for a given operation depend on the client-centric consistency setting applied, as well as dynamic parameters such as the current workload and network condition. We present OptCon, a machine learning-based predictive framework, that can automate the choice of client-centric consistency setting under user-specified latency and staleness thresholds given in the service level agreement (SLA). Under a given SLA, OptCon predicts a client-centric consistency setting that is matching , i.e., it is weak enough to satisfy the latency threshold, while being strong enough to satisfy the staleness threshold. While manually tuned consistency settings remain fixed unless explicitly reconfigured, OptCon tunes consistency settings on a per-operation basis with respect to changing workload and network state. Using decision tree learning, OptCon yields 0.14 cross validation error in predicting matching consistency settings under latency and staleness thresholds given in the SLA. We demonstrate experimentally that OptCon is at least as effective as any manually chosen consistency settings in adapting to the SLA thresholds for different use cases. We also demonstrate that OptCon adapts to variations in workload, whereas a given manually chosen fixed consistency setting satisfies the SLA only for a characteristic workload.

The exponential growth of interconnected systems across the Internet of Things (IoT), Software-Defined Networks (SDN), and cloud environments has led to a drastic increase in the scale and advancement of Distributed Denial of Service (DDoS) attacks. Conventional defences based on machine learning are often static, traffic-centric, and lack adaptivity to dynamic network behaviour, resulting in high false-positive rates and delay in mitigation responses. To address these challenges, this paper presents a conceptual framework for an AI-driven, context-aware DDoS detection and mitigation system. The proposed approach employs a optimized Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN–BiLSTM) model for efficient, real-time detection of DDoS attacks. The model captures spatial and temporal correlations in traffic while incorporating contextual parameters such as Quality of Service (QoS), Service Level Agreement (SLA) priority, and workload telemetry. These contextual attributes enable the model to distinguish legitimate network fluctuations from malicious activities, so that the false positive rate will be reduced without compromising accuracy. For adaptive response, a Reinforcement Learning (RL)-based mitigation layer—exploiting Deep Q-Network (DQN) and Actor–Critic (AC) variants—continuously learns optimal mitigation actions based on real-time feedback. The interaction between detection and mitigation components forms a context-aware feedback loop, enabling dynamic policy refinement and continuous system improvement. Comprehensive literature analysis reveals that existing AI-based DDoS defense mechanisms lack integration between detection and mitigation, and often overlook contextual intelligence. The proposed framework bridges this gap by introducing a self-adaptive, feedback-driven defense system capable of maintaining detection precision, low latency, and QoS preservation in heterogeneous environments.

This technical article critically examines the strengths and advantages of the modern data analytics platform in comparison to legacy ETL (Extract, Transform, Load) tools through an in-depth exploration of key features, performance, scalability, ease of use, and integration capabilities. This article will highlight how the advancement in technology has resulted in offering superior capabilities in terms of real-time processing, scalability, agility, advanced analytics, unified platforms, governance, security, deployment flexibility, cost efficiency, and ecosystem integration compared to traditional ETL processes. These advantages make them crucial for organizations aiming to maximize the value of their data assets and foster innovation in their business operations. There are diverse options available for implementing a new data pipeline or upgrading the existing orchestration mechanism. This article tends to do a thorough analysis by carefully evaluating these benefits, weaknesses, and considerations. Researchers and industry specialists can make informed decisions about selecting the appropriate ETL components or considering alternative data integration and processing approaches (such as ELT, streaming data platforms, or modern data analytics tools) that better suit their specific requirements and use cases.

The integration of artificial intelligence into data engineering processes represents a paradigmatic shift in how organizations manage, process, and derive value from their data assets. This comprehensive technical review examines the transformative impact of machine learning on traditional Extract, Transform, Load (ETL) workflows and data pipelines. Starting with intelligent data extraction capabilities that leverage natural language processing and computer vision, continuing through adaptive transformation logic and smart loading optimization, AI enhances every aspect of the data engineering lifecycle. Advanced anomaly detection and automated quality control mechanisms enable proactive identification of issues before they impact downstream systems. Reinforcement learning algorithms optimize resource allocation while self-tuning pipelines continuously refine operational parameters without human intervention. Despite significant benefits, organizations face substantial implementation challenges including explainability limitations, skills gaps, legacy system integration, and governance considerations. The emerging landscape features knowledge graphs for semantic understanding, generative AI for pipeline creation, and cross-organizational data fabrics with embedded intelligence innovations that collectively blur traditional boundaries between data engineering and data science disciplines.

Data pipelines play an important role throughout the data management process whether these are used for data analytics or machine learning. Data-driven organizations can make use of data pipelines for producing good quality data applications. Moreover, data pipelines ensure end-to-end velocity by automating the processes involved in extracting, transforming, combining, validating, and loading data for further analysis and visualization. However, the robustness of data pipelines is equally important since unhealthy data pipelines can add more noise to the input data. This paper identifies the essential elements for a robust data pipeline and analyses the trade-off between data pipeline robustness and complexity.

This paper presents a conceptual framework for enhancing data ingestion and ELT (Extract, Load, Transform) pipelines to support seamless digital transformation in cloud environments. As organizations increasingly rely on cloud technologies to manage and analyze vast amounts of data, optimizing data pipelines becomes critical for ensuring scalability, performance, and security. The proposed framework integrates four core components: An ingestion layer, a transformation engine, orchestration, and monitoring and governance. Each component is designed to address key challenges such as data latency, integration complexity, and compliance requirements, while ensuring robust data management practices. Through a combination of design science research, empirical case studies, and industry-based insights, this framework provides an adaptable and scalable solution for data-centric enterprises. The framework’s architecture emphasizes modularity, automation, resilience, and security, promoting efficient data processing workflows across various industries. The paper also discusses the practical implications of adopting the framework, highlighting its potential to optimize business intelligence, reduce operational costs, and enhance data governance. Lastly, the study identifies avenues for future research, particularly in integrating AI-driven optimizations and enhancing performance metrics. This paper contributes to the growing body of work on cloud-based data engineering and offers a comprehensive approach to tackling the complexities of modern data pipelines in a rapidly evolving digital landscape.

In the context of Industry 4.0, data pipelines are increasingly central to the integration of systems, decision support, and real-time industrial automation. One critical capability is understanding and tracing how data flows and transforms through ETL (Extract, Transform, Load) stages. Existing approaches provide limited semantic visibility, relying mostly on logs, static DAGs, or limited metadata tagging. This paper presents a practical and reproducible implementation of a semantic-aware data lineage tracking system, using graph modeling (Neo4j), ontology representation (RDF), and semantic querying (SPARQL). We detail the design, implementation, and experimental validation of the pipeline, using the AdventureWorks DW2022 dataset. Results show the effectiveness of the model in enhancing traceability, querying flexibility, and integration-readiness for industrial-scale deployments.

cryoem-cloud-tools: A software platform to deploy and manage cryo-EM jobs in the cloud.

Cloud computing has become an alternative IT infrastructure where users, infrastructure providers, and service providers all share and deploy resources for their business processes and applications. In order to deliver cloud services cost effectively, users' data is stored in a cloud where applications are able to perform requests from clients efficiently. As data is transferred to the cloud, data owners are concerned about the loss of control of their data and cloud service providers (CSPs) are concerned about their ability to protect data when it is moved about both within and out of its own environment. Many security and protection mechanisms have been proposed to protect cloud data by employing various policies, encryption techniques, and monitoring and auditing approaches. However, data is still exposed to potential disclosures and attacks if it is moved and located at another cloud where there is no equivalent security measure at visited sites. In a realistic cloud scenario with hierarchical service chain, the handling of data in a cloud can be delegated by a CSP to a subprovider or another. However, CSPs do not often deploy the same protection schemes. Movement of user's data is an important issue in cloud, and it has to be addressed to ensure the data is protected in an integrated manner regardless of its location in the environment. The user is concerned whether its data is located in locations covered by the service level agreement, and data operations are protected from unauthorized users. When user's data is moved to data centers located at locations different from its home, it is necessary to keep track of its locations and data operations. This chapter discusses data protection and mobility management issues in cloud environment and in particular the implementation of a trust-oriented data protection framework.

Light-sheet fluorescence imaging is a cutting-edge 3-dimensional imaging technique that allows the analysis of biological structures and processes in-vivo and in-vitro with exceptional image resolutions [1].In recent years, this technique has evolved to increase the microscope field of view and imaging throughput resulting in the generation of huge amounts of data in the range of hundreds of Gigabytes to Terabytes per specimen [2].Consequently, this surge in data volumes presents challenges related to storage, data organization, image processing, analysis, metadata management, and visualization.Thus far, laboratories have predominantly relied on on-premise computing resources and storage solutions for data processing.These solutions afford them complete control over data and resources, thereby facilitating superior latency in data accessibility and customized security measures.Nevertheless, this approach is fraught with limitations, including constrained flexibility of compute resources, external data accessibility, scalability, and disaster recovery capabilities [3].At the Allen Institute for Neural Dynamics, we are studying large populations of specimens and sharing the data to the scientific community as early as possible which led us to rely on cloud computing resources and cloud friendly formats (OMEZarr) to develop our image processing pipelines.We are engaged in the imaging process of whole mouse brains mounted on the SmartSPIM, a Selective Plane Illumination Microscope (SPIM), and commercial light-sheet microscope system developed by LifeCanvas Technologies.Imaging is usually conducted at a resolution of 2.0 m z step, with an x-y pixel size of 1.8 m.This system facilitates the acquisition of highresolution images through the utilization of various lasers and adjustable laser powers, enabling the generation of multiple channels for dataset acquisition.In this work, we introduce the SmartSPIM pipeline, an end-to-end automated cloud light-sheet image processing pipeline which uses Code Ocean and Amazon Web Services to efficiently handle vast quantities of data.To the best of our knowledge, it is the first automated cloud-based light-sheet image processing pipeline that has been successfully deployed for this purpose.This pipeline includes rapid data transfer from a high-performance on-premise storage system to the cloud, creation of metadata, flatfield correction, image correction to remove stripe noise, large-scale image stitching facilitated by TeraStitcher [4], image registration to the Allen Mouse Brain Common Coordinate Framework (CCF) [5], a cell localization module using Cellfinder [6], quantification of cell counts per CCF region, and a cloud-native visualization of the stitched 3D volume and cell distributions categorized by brain region with Neuroglancer.It is worth mentioning that raw data, code, and results are made fully accessible to the public as an integral component of fair, open and reproducible science.Our approach involves structuring our raw data with experiment metadata.Furthermore, the results of each processed dataset include metadata derived from both raw data and all subsequent image processing modules.This derived metadata comprehensively encompasses inputs, algorithmic parameters, utilized tools, timestamps, and software versions employed at each stage of the pipeline, facilitating future reproducibility efforts.Lastly, all our code is publicly available on Github.This pipeline is capable of concurrently processing multiple datasets.Typically, the dataset size varies between 1 and 2 Terabytes, contingent upon the number of channels and nature of the experiment.For instance, in the case of a dataset of 1.5 Terabytes and 3 channels, the processing times are currently as follows: 3.3 hours for applying flatfield correction and removing stripe noise, 0.2 hours for image stitching while exclusively considering image registration within overlapping image stacks in a single channel, 1.7 hours for fusing the data with OMEZarr format, 0.1 hours for registering a single channel to the CCF atlas, 5 hours for whole-brain cell localization in multiple channels, and 0.2 hours for mapping identified cells to each CCF region.

Cyber-Manufacturing System (CMS) is a vision for the factory of the future, where physical manufacturing resources and processes are integrated with computational workflows to provide on-demand, adaptive, and scalable manufacturing services. In CMS, functional manufacturing components in a factory floor are digitized and encapsulated in production services; and are accessible by users throughout the network. CMS utilizes data-centric technologies to program manufacturing activities in factory floors. Leveraging advanced technologies, CMS can provide robust solutions to achieve better manufacturing agility, flexibility, scalability, and sustainability than from traditional factories. While data is the main driver of the manufacturing activities in CMS, the lack of (i) a generic data model of explicit representation of the entities and stakeholders in CMS and (ii) workflow definition and analysis for service-orientated functionalities and manufacturing intelligence of CMS is still hindering the implementation of a fully executable CMS. To address such problems, this paper (i) formalizes a data modeling of CMS using Entity-Relationship (E-R) diagram, (ii) presents the definition and analysis of workflows along with data pipelines and Extract/Transform/Load (ETL) processes that automate the entire lifecycle activities in CMS and (iii) deploys the proposed data model and workflows in a Web-based application, and (iv) tests the functionality of this application with an industrial case and eventually validates the proposed data model and workflows.

With the rapid development of mobile edge computing (MEC) and unmanned aerial vehicle (UAV) communication networks, UAV-assisted edge computing has emerged as a promising paradigm for low-latency and energy-efficient computation. However, the time-varying nature of air-to-ground channels and the coupling between UAV trajectories and computation offloading decisions significantly increase system complexity. To address these challenges, this paper proposes an Adaptive UAV Edge Intelligence Framework (AUEIF) for joint UAV computation offloading and trajectory optimization under dynamic channels. Specifically, a dynamic graph-based system model is constructed to characterize the spatio-temporal correlation between UAV motion and channel variations. A hierarchical reinforcement learning-based optimization framework is developed, in which a high-level actor–critic module is responsible for generating coarse-grained UAV flight trajectories, while a low-level deep Q-network performs fine-grained optimization of task offloading ratios and computational resource allocation in real time. In addition, an adaptive channel prediction module leveraging long short-term memory (LSTM) networks is integrated to model temporal channel state transitions and to assist policy learning and updates. Extensive simulation results demonstrate that the proposed AUEIF achieves significant improvements in end-to-end latency, energy efficiency, and overall system stability compared with conventional deep reinforcement learning approaches and heuristic-based schemes while exhibiting strong robustness against dynamic and fluctuating wireless channel conditions.