Conquering big data with spark and BDAS

With the advancement in technology, a huge amount of data containing useful information, called Big Data, is generated on a daily basis. For processing such tremendous volume of data, there is a need of Big Data frameworks such as Hadoop MapReduce, Apache Spark etc. Among these, Apache Spark performs up to 100 times faster than conventional frameworks like Hadoop Mapreduce. For the effective analysis and interpretation of this data, scalable Machine Learning methods are required to overcome the space and time bottlenecks. Partitional clustering algorithms are widely adopted by researchers for clustering large datasets due to their low computational requirements. Thus, we focus on the design of partitional clustering algorithm and its implementation on Apache Spark. In this paper, we propose a partitional based clustering algorithm called Scalable Random Sampling with Iterative Optimization Fuzzy c-Means algorithm (SRSIO-FCM) which is implemented on Apache Spark to handle the challenges associated with Big Data Clustering. Experimentation is performed on several big datasets to show the effectiveness of SRSIO-FCM in comparison with a proposed scalable version of the Literal Fuzzy c-Means (LFCM) called SLFCM implemented on Apache Spark. The comparative results are reported in terms of value of F-measure, ARI, Objective function, Run-time and Scalability. The reported results show the great potential of SRSIO-FCM for Big Data clustering.

Purpose Big data analytics allows researchers and industry practitioners to extract hidden patterns or discover new information and knowledge from big data. Although artificial intelligence (AI) is one of the emerging big data analytics techniques, hospitality and tourism literature has shown minimal efforts to process and analyze big hospitality data through AI. Thus, this study aims to develop and compare prediction models for review helpfulness using machine learning (ML) algorithms to analyze big restaurant data. Design/methodology/approach The study analyzed 1,483,858 restaurant reviews collected from Yelp.com. After a thorough literature review, the study identified and added to the prediction model 4 attributes containing 11 key determinants of review helpfulness. Four ML algorithms, namely, multivariate linear regression, random forest, support vector machine regression and extreme gradient boosting (XGBoost), were used to find a better prediction model for customer decision-making. Findings By comparing the performance metrics, the current study found that XGBoost was the best model to predict review helpfulness among selected popular ML algorithms. Results revealed that attributes regarding a reviewer’s credibility were fundamental factors determining a review’s helpfulness. Review helpfulness even valued credibility over ratings or linguistic contents such as sentiment and subjectivity. Practical implications The current study helps restaurant operators to attract customers by predicting review helpfulness through ML-based predictive modeling and presenting potential helpful reviews based on critical attributes including review, reviewer, restaurant and linguistic content. Using AI, online review platforms and restaurant websites can enhance customers’ attitude and purchase decision-making by reducing information overload and search cost and highlighting the most crucial review helpfulness features and user-friendly automated search results. Originality/value To the best of the authors’ knowledge, the current study is the first to develop a prediction model of review helpfulness and reveal essential factors for helpful reviews. Furthermore, the study presents a state-of-the-art ML model that surpasses the conventional models’ prediction accuracy. The findings will improve practitioners’ marketing strategies by focusing on factors that influence customers’ decision-making.

Big data and data analytics are currently the buzzwords in both academia and industry to become data driven. Big data has been the trending topic in the accounting industry also. Big data and data analytics will have an important impact on accounting and accountants. Big data will improve the quality of accounting information and the accounting profession will continue to provide real-time and dynamic information to assist in decision-making. The purpose of this research is to investigate the impact of big data and big data analytics in accounting. Data analytics is one of the most recent developments in the accounting context. This study is qualitative in nature and adopted a literature review methodology to gain a better understanding of the study area. This literature review seeks to provide a description and evaluation of the impact of big data analytics on accounting. This research found that big data presents great opportunities for decision making in accounting and risks analysis, which indicated that companies could improve their performance, measure performance, manage risks and allow effective real-time decision-making with data analytics. This research revealed that accountants can create more value in a world of big data analytics and encourages accountants to get started with big data to find answers to risks in business operations as well as understand financial performance. It shows that relying on big data analytics will open new possibilities for accountants. This study contributes to the research literature in the area of big data analytics on accounting. The limitations of this study are that it utilizes few recent peer reviewed articles in the general accounting practice, therefore not exhaustive in describing how big data and big data analytics impacts accounting.

Embedded industries have invested a lot in the introduction of software product lines in their software development. In addition, using open source software appears to be a profitable way to obtain good software. This is also applicable for organizations doing product line engineering. On the other hand, because of the diverse use of open source software, product line development is an attractive way of working in open source communities. In fact, the configuration mechanisms used in open source communities may be applicable within software product lines as well. In addition, product line organizations are usually involved in distributed development, which works very efficiently within open source communities. However, at present, there is limited interaction between the open source and product line development communities. Our aim is to explore what the two communities can learn from each other and to develop a better understanding of how the two communities can benefit from each other.

Cloud computing and big data: Technologies and applications

This work was partially funded by the Spanish Ministry of Economy, Industry and Competitiveness under the grant TIN2016-79637-P ”Towards Unification of HPC and Big Data Paradigms”; the Spanish Ministry of Education under the FPU15/00422 Training Program for Academic and Teaching Staff Grant; the Advanced Scientific Computing
\nResearch, Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357; and by DOE with agreement No. DE-DC000122495, program manager Laura Biven.

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

Big data applications in healthcare have provided a variety of solutions to reduce costs, errors, and waste. This work aims to develop a real-time system based on big medical data processing in the cloud for the prediction of health issues. In the proposed scalable system, medical parameters are sent to Apache Spark to extract attributes from data and apply the proposed machine learning algorithm. In this way, healthcare risks can be predicted and sent as alerts and recommendations to users and healthcare providers. The proposed work also aims to provide an effective recommendation system by using streaming medical data, historical data on a user’s profile, and a knowledge database to make the most appropriate real-time recommendations and alerts based on the sensor’s measurements. This proposed scalable system works by tweeting the health status attributes of users. Their cloud profile receives the streaming healthcare data in real time by extracting the health attributes via a machine learning prediction algorithm to predict the users’ health status. Subsequently, their status can be sent on demand to healthcare providers. Therefore, machine learning algorithms can be applied to stream health care data from wearables and provide users with insights into their health status. These algorithms can help healthcare providers and individuals focus on health risks and health status changes and consequently improve the quality of life.

The unprecedented proliferation of miniaturized sensors and intelligent communication, computing, and control technologies have paved the way for the development of the Industrial Internet of Things. The IIoT incorporates machine learning and massively parallel distributed systems such as clouds, clusters, and grids for big data storage, processing, and analytics. In IIoT, end devices continuously generate and transmit data streams, resulting in increased network traffic between device-cloud communication. Moreover, it increases in-network data transmissions. requiring additional efforts for big data processing, management, and analytics. To cope with these engendered issues, this article first introduces a novel concentric computing model (CCM) paradigm composed of sensing systems, outer and inner gateway processors, and central processors (outer and inner) for the deployment of big data analytics applications in IIoT. Second, we investigate, highlight, and report recent research efforts directed at the IIoT paradigm with respect to big data analytics. Third, we identify and discuss indispensable challenges that remain to be addressed for employing CCM in the IIoT paradigm. Lastly, we provide several future research directions (e.g., real-time data analytics, data integration, transmission of meaningful data, edge analytics, real-time fusion of streaming data, and security and privacy).

Integration of Spark framework in Supply Chain Management

The average volume of data produced daily is estimated to be over 2.5 quintillion byte. Moreover, by year 2020, it is estimated that 1.79MB of data will be created every second by each person in the world. Apparently, big datasets contain tremendous amount of valuable information that can be used for improved decision making. However, big data requires incredible amount of storage and computational resources for effective processing. Machine Learning (ML) algorithms are effective tools popularly used to analyze and extract concealed insights from datasets. However, some ML algorithms were not originally designed to handle big datasets, hence their computational complexity decreases with increase in data size. Consequently, this makes big data analytics extremely slow or unrealistic. Therefore, there is an obvious need for fast and effective techniques for big data analytics. This paper introduces an intelligent hybrid ML-based technique suitable for big data analytics (called EDISA_ML). EDISA_ML is a boundary detection and instance selection algorithm, inspired by edge detection in image processing. It was evaluated on four ML algorithms and big datasets, and the results show that it achieved a storage reduction of over 50% and simultaneously improved the training speed of the evaluated ML algorithms by over 93% (in some cases), without meaningfully affecting their prediction accuracy.

Nowadays, Next Generation Sequeencing (NGS) is a catch-all term used to describe different modern DNA sequencing applications that produce big genomics data that can be analysed in a faster fashion than in the past. For this reason, NGS requires more and more sophisticated algorithms and high-performance parallel processing systems able to analyse and extract knowledge from a huge amount of genomics and molecular data. In this context, researchers are beginning to look at emerging deep learning algorithms able to perform efficient big data analytics. In this paper, we analyse and classify the major current deep learning solutions that allow biotechnology researchers to perform big genomics data analytics. Moreover, by means of a taxonomic analysis, we provide a clear picture of the current state of the art also discussing future challenges.

Massive amount of data in different forms need to be handled in any healthcare applications. Type of data, size of data, data security and other features has more significance in handling the data. The term big data refers to data with certain characteristics, volume, velocity, value, veracity and variability. Such big data need to be stored, processed, and analyzed for required results. Medical data has more complexity in predicting the results from it, which will have more significance in patient's treatment. Because of its significance, there is need of developing efficient and better performing algorithms, techniques and tools to analyze medical big data. Whereas, the traditional algorithms are not capable for analyzing such complex data. Machine learning algorithms well fit for these kinds of data and analytics. In this Keywords: Big data, Health care, disease prediction, SVM, CNN survey paper, we discussed about characteristic of big data, features of big data, how to represent big data, different types of machine learning algorithms used in big data analytics. We discussed about big data analytics in major healthcare areas like EHR maintenance, disease diagnose, prediction of emergency condition of patients, etc.,. Also stated different machine algorithms usage in disease diagnose and patient's data analysis and discussed about importance of various machine learning algorithms. Here, we have highlighted the areas where big data analytics have been applied in healthcare sectors. It describes the characteristics and features of big data, importance of big data analytics in healthcare sectors, various machine learning algorithms used in big data analytics and their efficiency.

Resting state functional magnetic resonance imaging (rfMRI) can be used to measure functional connectivity and then identify brain networks and related brain disorders and diseases. To explore these complex networks, however, huge amounts of data are necessary. Recent advances in neuroimaging technologies, and the unique methodological approach of rfMRI, have enabled us to an era of Biomedical Big Data. The recent progress of big data sharing projects with their challenges are discussed. This increasing amount of neuroimaging data has greatly increased the importance of developing preprocessing pipelines and advanced analytic techniques, which are better at handling large-scale datasets. Before applying any analysis method on rfMRI data, several preprocessing steps need to be applied to reduce all unwanted effects. Three alternative ways to get access to big preprocessed rfMRI data are presented involving the minimal preprocessing pipelines. There are several commonly used methods to examine functional connectivity. However, they become limited in the analysis of big data, and a new tool to explore such data is necessary. We propose a number of novel methods rooted in algebraic topology and collectively referred to as Topological Data Analysis to rfMRI functional connectivity. Their properties for big data analysis are also discussed.

Adding big data to biology is changing the way we describe how cells work, which lets us make more complete and accurate models of biological systems. With the rise of high-throughput technologies like genetics, transcriptomics, proteomics, and metabolomics, huge amounts of biological data are being produced, which gives scientists a chance to study how cells work like never before. This essay looks at how big data can help us better understand how cells work by focussing on how machine learning, network analysis, and systems biology can be used. Patterns can be found in multi-omic datasets using machine learning algorithms, such as guided and uncontrolled learning. These algorithms help us understand gene regulation, protein interactions, and cellular reactions. Network-based methods improve cellular process modelling even more by building complex networks of molecule interactions that mirror the complicated control systems inside cells. Systems biology and computer simulations provide dynamic models that duplicate how cells behave, which makes it easier to study how diseases work, how drugs work, and how to treat them. Even with these improvements, there are still problems with integrating, scaling, and interpreting data because living systems are very complicated and data is not all the same. The study also talks about how artificial intelligence (AI) can help solve these problems, make predictions more accurate, and make cellular models more useful. Using big data modelling in disease research, like studying cancer and brain illnesses, shows that it can help find signs, make treatment plans better, and move personalised medicine forward. In the end, combining big data with bioinformatics has the huge potential to change how we think about how cells work, opening up new areas for medical study and drug creation.