• Home
  • Search
  • AI-powered data engineering: How machine learning is revolutionizing ETL and data pipelines
  • https://doi.org/10.30574/wjaets.2025.15.3.0858Copy DOI Icon

AI-powered data engineering: How machine learning is revolutionizing ETL and data pipelines

  • Abstract
  • Literature Map
  • Similar Papers
Abstract

The integration of artificial intelligence into data engineering processes represents a paradigmatic shift in how organizations manage, process, and derive value from their data assets. This comprehensive technical review examines the transformative impact of machine learning on traditional Extract, Transform, Load (ETL) workflows and data pipelines. Starting with intelligent data extraction capabilities that leverage natural language processing and computer vision, continuing through adaptive transformation logic and smart loading optimization, AI enhances every aspect of the data engineering lifecycle. Advanced anomaly detection and automated quality control mechanisms enable proactive identification of issues before they impact downstream systems. Reinforcement learning algorithms optimize resource allocation while self-tuning pipelines continuously refine operational parameters without human intervention. Despite significant benefits, organizations face substantial implementation challenges including explainability limitations, skills gaps, legacy system integration, and governance considerations. The emerging landscape features knowledge graphs for semantic understanding, generative AI for pipeline creation, and cross-organizational data fabrics with embedded intelligence innovations that collectively blur traditional boundaries between data engineering and data science disciplines.

Similar Papers
  • Research Article

Integration of Ground-Penetrating Radar and Artificial Intelligence for Soil Piping- A Review

  • Apr 02, 2024
  • Journal of Applied Science, Engineering, Technology and Management
  • Anju R Nair +1
  • Research Article
  • Citations15

Machine Learning And Artificial Intelligence in Diabetes Prediction And Management: A Comprehensive Review of Models

  • Dec 17, 2024
  • Innovatech Engineering Journal
  • Md Ashraful Alam +3
  • Abstract

SAT-469 Update on the Use of Robotic Surgery and Artificial Intelligence for Adrenal Tumors

  • Oct 22, 2025
  • Journal of the Endocrine Society
  • Hassan M Heshmati +1
  • Research Article

Trends of recent data, AI/ML approaches for geospatial AI in Earth observation towards sustainable development goals

  • Mar 11, 2025
  • ITU Journal on Future and Evolving Technologies
  • Shivangi Somvanshi +2
  • Research Article

Optimizing Staff Recruitment and Selection Processes with Artificial Intelligence Technologies

  • Jan 01, 2025
  • Экономические науки
  • Антон Олегович Пилипчук +1
  • Research Article
  • Citations1

Robotic Process Automation to Autonomous Bots: A Survey on the Integration of Artificial Intelligence into Robotic Process Automation

  • Jan 01, 2025
  • Scientiarum: A multidisciplinary journal
  • Essakki Gomathy P +1
  • Book Chapter

Smart Dining

  • Jul 18, 2025
  • R Siva Subramanian +6
  • Research Article
  • Citations1

Integration of Artificial Intelligence in Industrial Education: A Review of Current Trends and Future Directions

  • Aug 31, 2025
  • Journal of Computer, Software, and Program
  • Agbomehre M Momoh +3
  • Research Article

Role of AI in Enhancing Critical Thinking in Science Education: Challenges and Opportunities for Science Instructor

  • Aug 23, 2025
  • International Journal of Multidisciplinary: Applied Business and Education Research
  • Charlie T Anselmo +5
  • Research Article

The Influence of Artificial Intelligence on Teacher Professional Identity and Job Satisfaction

  • Dec 13, 2025
  • The Critical Review of Social Sciences Studies
  • Zarina Naz +3
  • Research Article

Artificial Intelligence and Machine Learning Applications in the Postharvest Storage of Tomato

  • Jun 02, 2025
  • Asian Journal of Agricultural and Horticultural Research
  • Elvan Ekinci +1
  • Book Chapter

AI in B2B Marketing and Its Applications in the Hospitality Industry

  • Apr 30, 2025
  • Sonal Jain
  • Research Article

Reimagining Library Services: The Role of Artificial Intelligence in Modern Library Management

  • Aug 07, 2025
  • International Scientific Journal of Engineering and Management
  • Dr Sachin Suryawanshi +2
  • Research Article

Revolutionizing enterprise resource planning: The integration of Artificial Intelligence and machine learning in SAP ecosystem transformation

  • Jul 30, 2019
  • World Journal of Advanced Research and Reviews
  • Venu Gopal Avula
  • Research Article

Conceptual Framework for Training Specialists for the Maritime Transport Industry

  • Mar 04, 2026
  • Siberian Pedagogical Journal
  • Evgeny M Dorofeev
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.