Challenges and Techniques for Testing of Big Data

Big Data - Towards a New Techno-Determinism?

Artificial intelligence (AI) and machine learning (ML) have transformed business analytics in today’s data-driven environment, bringing new trends, technologies, and capabilities. Focusing on areas such as machine learning, predictive analytics, natural language processing (NLP), text analytics, computer vision, and robotic process automation (RPA), this research examines the AI-based technologies needed to advance business intelligence (BI). By integrating these solutions, organizations can improve operations, gain valuable insights, and make data-driven decisions. Finding patterns and predicting trends in big data is accomplished through machine learning and predictive analytics. Natural Language Processing (NLP) and text analytics help unlock the potential of redundant data by facilitating user insights, service automation, and better decision making. This expands business intelligence to include visual data processing. Also, use RPA and AI to automate processes to increase productivity and efficiency. Big data and IoT can also help businesses better adapt to changing business conditions. Advanced design techniques enhance human-computer interaction and enable intelligent data interpretation. This article explores the future potential of AI-ML NLP to reinvent analytics and decision making by highlighting the latest developments and impacts of AI-ML NLP on BI.

Big Data refers to complex and unstructured data that is difficult to analyse and utilize with traditional applications and analyses. Big Data comes from a variety of sources, including tracking and sensor devices which are widely used in logistics and supply chain management, and relate to Radio Frequency Identification (RFID) technology. Thus, this chapter reviews the literature on RFID adoption in supply chain/logistics management from 1995-2015. We identify current trends in the literature, drawing on the three levels of decision making, that is, strategic, tactical, and operational. We suggest that more research needs to be conducted with regards to the intangible benefits of RFID, the use of RFID big data for achieving higher performance, and to shift the focus from the ‘what' and the impacts on performance to the ‘how' and the ways RFID is adopted and assimilated in organizations and supply chains. Finally, the managerial implications of our review as well as the limitations and future research directions are outlined.

Big data has evolved as a most challenging area in scientific study and research. It has drawn much attention during the last few years. It influences our modern society, business, government, healthcare, research and almost every discipline. In this data-driven era, where data is continuously acquired from a verity of sources for different purposes, the ability to make timely decisions based on available data is a very critical task. The massive data size, variety, velocity, accuracy and high dimensionality presents new challenges to big data. This paper attempts to present some challenges of big data. In addition, a study on the conceptual design of big data architecture presented on specific big data applications. In this paper, we presented research work on big data analytics techniques in the areas of text, sentiment, video, social media and predictive analytics. A comparative analysis on selected big data applications has been presented in detail.

Big Data is a collection of large amount of data which is growing very rapidly with the popularity of social networking sites. The size of the Big data has been extended from terabytes to petabytes. Big data are characterized by four important attributes: volume, velocity, variety and veracity. The volume attributes describe the data at rest in the range from terabytes to Exabyte's, the velocity deals with data in motion, i.e streaming the data to respond within milliseconds rather than in seconds, the variety discuss the data in many different forms such as structured, unstructured, text and multimedia data, whereas the veracity deals with the data in doubt, i.e. uncertainty in data due to data inconsistency. These attributes of big data make it a challenge for organizations to have control over, and use such data. In today's era we are overloaded with the information, however we are lacking the insight. The 90% of the data in the world today has been created in the last two years by various social networking sites. As big data grows it challenges the capabilities of traditional data warehouses that collect and store large amounts of internal and external data. Data drawn from these repositories are used to improve decision making, increase organizational efficiencies, and improve organizational effectiveness. In this paper an attempt has been made to explore the real life application of Big Data , cloud database, Hadoop, Map Reduce and Cloud Computing, in various domains.

The development of mobile Internet, Internet of Things, and cloud computing has contributed to the unprecedented growth of information data. Big data plays a very important role in education. Currently, the literature review and in-depth research on big educational data are not very extensive, mainly involved in two fields: education mining and learning analysis. For a perfect research about education big data, this paper comprehensively reviewed three major aspects (Predictive Analytics, Learning Analytics, and Recommendation Systems) of educational data analytics for an intensive investigation and analysis: (1) Predictive Analytics: It predicts students’ learning performance by tracking students’ learning information and then analyzes students’ learning competence to build an academic early warning system; teachers can be allowed to intervene in students in time and adopts different teaching ways for different students. Therefore, both students’ learning and ability can be individualized and improved; (2) Learning Analytics: This part can identify the learners’ behavior patterns and obtain more implicit learner characteristics by studying the hidden meaning behind learning behaviors and strategies; (3) Recommendation Systems: It can match the needs of learners and recommend appropriate learning resources through different methods. All the above proved that the application of big data technology in education provides powerful data support for the development of education.

This paper argues that mimicry is a central issue in the development of new practices of predictive analytics and big data. The issue concerns the increasingly precise reproduction of human interactional dynamics and their translation to machine and code worlds. Mimicry, in this sense, allows for predictive analytics to simulate a huge range of individual and collective cross-species behaviours. This includes human practices, particularly at the non-conscious, non-verbal level sometimes with an uncanny appearance of intuitive anticipation. The paper takes up different perspectives on mimesis and simulation to discuss the exploitative and emancipatory tensions these articulate. It investigates these through developments in swarm robotics, text mining and recent advances in codifying human synchrony. All of these utilise mimicry as core elements in their development.

Recent years have witnessed a revolution in data science and ‘big data’ into which psychiatric research has also being drawn. ‘Big data’ has been defined as data sets which are so large in size, so fast to change, and so complex in structure that traditional data processing techniques are overwhelmed [1]. The mining and exploitation of such big data resources as Electronic Healthcare Records (EHRs) present an exciting challenge to the field of psychiatric epidemiology. The number of big data projects within psychiatric research is growing, and Stewart and Davis’s literature review is, therefore, timely [2]. Technological advances in data processing and storage, computer networking, mobile technology, and data manipulation have rendered huge quantities of healthcarerelated data potentially amenable to analysis. Such studies potentially offer much larger patient numbers, wider parameters of study, and longer timescales of follow-up, than is typical of randomised controlled trials (RCTs) or cohort studies. A further advantage of this data is it often arises from naturalistic clinical settings, in terms of both clinical practice and patient health and comorbidity. Indeed, while often considered the ‘gold standard’ in medical research, RCTs do have important limitations, such as overly-strict exclusion criteria. Routine clinical data sets can, therefore, be complementary to RCT data, while also making research findings more relevant to everyday clinical practice. In addition, the quantity of clinical ‘big data’ potentially allows analysis of rarer clinical conditions, or subject areas that would be unlikely to meet ethical approval for more conventional studies (for example, medication usage in pregnancy). Big data studies provide the scale and breadth of patient numbers required for stratified, predictive, and personalised medicine research. In addition, notwithstanding many challenges of working with big data, large-scale analysis of routinely collected healthcare data sets has already demonstrated effectiveness in fields such as pharmacovigilance [3] and post-marketing clinical trials. As described in detail by Stewart and Davis, the challenging aspects of working with ‘big data’ are captured by the taxonomy of ‘Vs’—volume, velocity, and variety—first described by Laney [4] and extended since [1, 2]. There are additional issues to working with healthcare big data specifically, including that clinical administrative data are generally not collected, curated, or formatted in a manner optimised for research, and the inherent sensitivity of the data in terms of personal privacy. As the possibilities for healthcare big data expand into such areas as text mining and natural language processing of clinical records, near ‘real-time’ updating of repositories, and incorporation of new data streams from mobile and wearable technology, robust systems must be in place to channel the potential data deluge. How great a role ‘cloud’ computing and storage will have in this is an intriguing question. In healthcare big data, the benefits offered by the cloud in increased power and accessibility, and reduced cost, must be weighed against concerns over data ownership, encryption, and unauthorized access. An additional question is whether unique patient identifiers (UPIs) utilised by some national healthcare systems can be extended more This comment refers to the article available at doi:10.1007/s00127016-1266-8.

Reservoir engineering is a critical discipline in the oil and gas industry, aimed at optimizing the extraction and management of subsurface resources. Traditionally, reservoir models have relied on static data and conventional simulation methods, which often fall short in capturing the dynamic complexities of reservoirs. With the advent of big data and predictive analytics, there is a significant opportunity to revolutionize reservoir modeling by integrating real-time data for continuous updates and more accurate predictions. This review introduces a theoretical exploration of how big data and predictive analytics can transform dynamic reservoir models, highlighting their potential to improve reservoir management and hydrocarbon recovery. Big data technologies enable the collection, storage, and processing of vast amounts of structured and unstructured data from multiple sources, including well logs, seismic surveys, drilling operations, and production data. Predictive analytics, through machine learning and statistical techniques, can extract actionable insights from these datasets to predict reservoir behavior, enhance decision-making, and optimize production strategies. This integration facilitates real-time monitoring and adaptive reservoir management, allowing engineers to respond to changing reservoir conditions and uncertainties more effectively. The review further explores the implications of big data-driven predictive models for enhanced hydrocarbon recovery techniques, such as water flooding and gas injection. By automating data processing and integrating real-time field data, predictive analytics can improve the accuracy of reservoir simulations, reduce downtime, and increase operational efficiency. Ultimately, the future of dynamic reservoir modeling lies in the seamless integration of big data and predictive analytics, providing a pathway to smarter, more sustainable resource management and maximized recovery factors in increasingly complex reservoir environments. Keywords: Big Data, Predictive Analytics, Reservoir Engineering, Dynamic Reservoir, Models.

The use of ‘big data’ and predictive analytics has transformed a sizeable number of industries, from insurance companies to print and e-book online retailers, and both the mass media and the scholarly literature have covered these developments. Online retailers have used big data systems to capture tremendous amounts of data about consumers and their purchases, which has enabled them to use predictive analytics and collaborative filtering systems to make purchase suggestions to consumers. Unfortunately, many university presses—even the largest presses with substantial endowments—have not been able to capitalize on the formidable marketing assets offered by big data and predictive analytics. In this article, the authors review the published literature and significant data sets, and present suggestions for the Canadian and US university press community to launch a non-profit direct-to-consumer Web site generating continuous-time sales and marketing data.

A massive bulk of data is being created due to digitalisation in various industries, including medical, manufacturing, sales, internet of things (IoT) devices, the web, and businesses. To find data patterns for data attributes machine learning (ML) algorithms are used. In this fast-growing world, we can see that data is generated in abundance by people, machines, and corporations. With the increase in computer science market, researchers are integrating heterogeneous and diverse data into accurate patterns by applying machine learning algorithms and complex strategies on data sets. The overabundance of high-dimensional big data has made it more difficult for scientists to extract important information from these data efficiently. Conventional data mining approaches are ineffective when dealing with large amounts of data. As big data increase exponentially, predictive analytics has become widely known. To evaluate a large number of data patterns, data driven technology predictive big data analytics (PBA) can be used and ML algorithms to investigate the present and future data based on the records of data patterns. In this research paper, predictive analysis on big data has been proposed using the splitting random forest (SRF) methodology with help of hyperparameter optimization and dimension reduction technique.

Visualization of big data is crucial for meaningful interpretations and especially for healthcare. Brief discussions are made for big data, background for healthcare, and recent work in big data analytics for healthcare. This research pertains to different levels of big data: 5,110 vs. 101,766 vs. 320,200 vs. 1 million data values. Data visualizations and predictive analytics are presented of big data for selected diseases of stroke with 5,110 data values, diabetes with 101,766 data values, and two COVID-19 studies: one with 320,200 data values and another with 1 million data values. Data visualizations are generated for these diseases with big data using Tableau. For stroke patients, an investigation was performed to determine how different living environments affect relationship between strokes. The data visualizations for diabetes showed impact of insulin use yielded reduced hospital stays. Data visualizations for COVID-19 provided temporal trends in confirmed cases, mortality, and recovery rates for 2020-2023. Conclusions and future directions of research are presented.

The rapid advancements in Artificial Intelligence (AI) and Big Data Analytics have significantly transformed financial markets, enabling more precise, data-driven, and strategic decision-making. This study explores the profound impact of AI-driven analytics in financial decision-making, focusing on how machine learning algorithms, predictive analytics, and automation enhance market efficiency, risk assessment, and investment strategies. The research highlights how AI-powered Big Data Analytics processes vast amounts of structured and unstructured financial data to identify patterns, trends, and anomalies that influence market movements. It further examines how financial institutions leverage AI for fraud detection, sentiment analysis, and algorithmic trading, leading to more informed and efficient investment decisions. Through a survey-based approach, this study collects data from financial professionals, analysts, and investors to evaluate their perspectives on AI’s role in improving data accuracy, reducing uncertainty, and optimizing financial strategies. The findings provide valuable insights into how AI-driven analytics contributes to minimizing risks, enhancing forecasting accuracy, and supporting regulatory compliance in financial markets. The study concludes that the integration of AI in Big Data Analytics is revolutionizing financial decision-making by improving efficiency, reducing human biases, and enabling real-time strategic actions. However, challenges such as data privacy concerns, ethical considerations, and the need for advanced AI governance frameworks remain crucial areas for future exploration. Keywords—Artificial Intelligence, Big Data Analytics, Financial Markets, Strategic Decision-Making, Algorithmic Trading, Risk Assessment, Predictive Analytics.

Predictive Analytics is of great interest when it comes to enhancing Business Intelligence. Businesses have already started to use Big Data Analytics, particularly predictive and prescriptive analytics, to strengthen and increase their business yields. Not only has analytics resulted in business growth, but has also provided a significant competitive edge over others. The voluminous data generated from various resources is highly unstructured in nature and adding a structure to it would leverage the actual potential of the data. New techniques and frameworks should serve as human aids in automatically and intelligently analyzing large datasets in order to acquire useful information. In this paper, we attempt to perform Big Data Analytics on data from one of the most important and growing sources, namely, Telecommunication. To keep pace with the growing telecommunication market and ever increasing demands of the consumers for quality service, the telecom service providers are required to observe and estimate various trends in customer's usage to plan future upgrades and deployments driven by real data. We have attempted to use several data mining techniques to find hidden and interesting patterns from the telecom data generated by Telecoms Italia cellular network for the city of Milano, Italy. K-means clustering is used to categorize the usage statistics while several machine learning algorithms like Decision Tree, Random Forest, Logistic Regression and SVM are used for predicting the usage of telecom services. In the end, a performance comparison matrix is generated to rate the performance of these algorithms for the given dataset. All these experiments are performed on the big data environment set up at the supercomputing infrastructure of C-DAC. Given such a matrix, the result can be applied to similar dataset pertaining to other domains as well.

The paper aims to contribute to the understanding of the connection of law and legal science, on the one hand, and the Big Data phenomenon, on the other. The connection of Big Data and law can be thematised in several ways. This article makes a distinction whereby there are two levels of interplay between Big Data and the law (and legal science). Big Data on the one hand can be the subject of legal regulation and legal science, but it also can be a tool for better, ‘predictive’ law making and lawyering. This latter is also true for legal science: Big Data opens a whole range of possibilities as a new tool. Thus, this article discusses three fields and questions in three sections: 1. Big Data as the subject of legal regulation. What kind of moral questions does Big Data, and the predictive potential it has, raise? How does law recently frame, define and regulate the Big Data phenomenon? How does Big Data affect existing legal framework rules regarding privacy, data protection, competition, business regulatory, etc.? What will the new rules, regulating Big Data look like? 2. Big Data as a tool in the regulator’s and the lawyer’s hand. How can we exploit the new possibilities provided by Big Data in law making, policy creation and the application of law? How can we design new ways of ‘Big Data-based social engineering’? How can we create predictive tools and inferencing techniques based on Big Data in policing, law enforcement and litigation? Finally in part 3. I discuss the impact of Big Data on legal science. How can Big Data, as a research tool help legal science? How do we use legal data-sets and textual corpuses as BD? How will these ‘super-empirical’ research methods affect legal scholarship? What is the relationship between traditional doctrinal scholarship and the new types of BD-based research? How can we use statistical analysis, natural language processing, content analysis, machine learning, behavioural prediction, etc. in legal science?