Design Approach to Big data Systems in Developing and Maintaining the Information Security Systems

Big Sensor Data Applications in Urban Environments

The emergence and development of big data tools, techniques and systems motivate industries and organizations to embrace and explore research in big data. This is to circumvent the challenges of the traditional database systems. However, the available benchmarks and workloads are for some specific aspect of Information Technology industries, which has differences in nature and complexities with the data obtained from other sources. Hence there is need for using data from other domains in order to evaluate the performance and maturity of the big data technologies. In this paper the performance profiling of Meteorological and Oceanographic data on Hive is conducted. Hive being the commonly used data warehouse analytical platform for big data is chosen with the view to exposing the intricacies that are involved in the formating and loading of the data. The response time for indexed and non-indexed retrievals using three set of queries frequently used in the area is found. The query types are Type 1 SELECT with WHERE clause, Type 2 SELECT with JOIN clause. And Type 3 SELECT with GROUP BY clause. The experimental results show that a good response time for both indexed and Non-indexed tables are achieved. The indexed retrieval shows a significant decrease in the response time for Type 1 query for all data sizes and for Type 3 query for 100GB data size and less . It also shows additional overhead for Type 2 query for all data sizes and Type 3 query for 500GB and more data sizes. The Meteorological and Oceanographic data if properly formated it's analytics with Hive proved to be efficient compared to the traditional database systems. The results of this study has the potentials of attracting the oil and gas companies to adopt big data technologies for the handling of their exploration dataset.

Cloud computing and big data: Technologies and applications

Internet of Things (IoT) and Big Data have fueled the development of fully distributed computational architectures for future cyber systems from data analytics, machine learning (ML) to artificial intelligence (AI). Trust and Privacy become two vital and necessary measures for distributed management of IoT powered big data learning systems and services. However, these two measures have been studied independently in computer science, social science and law.

Data is becoming the world's new natural resource and big data use grows quickly. The trend of computing technology is that everything is merged into the Internet and 'big data' are integrated to comprise complete information for collective intelligence. With the increasing size of big data, refining big data themselves to reduce data size while keeping critical data (or useful information) is a new approach direction. In this paper, we provide a novel data consumption model, which separates the consumption of data from the raw data, and thus enable cloud computing for big data applications. We define a new Data-as-a-Product (DaaP) concept, a data product is a small sized summary of the original data and can directly answer users' queries. Thus, we separate the mining of big data into two classes of processing modules: the refine modules to change raw big data into small sized data products, and application-oriented mining modules to discover desired knowledge further for applications from well-defined data products. Our practices of mining big stream data, including medical sensor stream data, streams of text data and trajectory data, demonstrated the efficiency and precision of our DaaP model for answering users' queries.

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.

Big data deals with data of very large data size, heterogeneous data types and from different sources. The data is very complex in nature and having growing data. Dealing with big data is one of the emerging areas of research which is expanding at a rapid rate in all domains of engineering and medical sciences. A major challenge imposes on the analysis of big data is originated from big data generation source, which generate data with very fast speed with varying data distribution due to which the classical methods are unable to process big data. This paper discusses the characteristics, challenges, and issues with big data mining. It also illustrates the examples taken from various fields like medical, finance, social networking sites, stock exchange, etc. to realize the application and importance of big data mining. This paper explains about the use of parallel computing in data mining security issues and how to deal with them. Furthermore, this paper also discusses challenges associated with big streaming data with concept drifts.

The traditional Internet of Brain Things’ big data information filtering method ignores the extraction of big data features, the filtering effect, and the effect of denoising processing on the simulation results, resulting in low filtering accuracy and poor performance. An effective information filtering and mining algorithm for the Internet of Brain Things based on support vector machine (SVM) is proposed. First, the model construction and feature extraction of the Internet of Brain Things’ big data system are carried out. The correlation feature extraction is performed on the effective information features; the correlation factors of the effective information data are sorted; the main feature quantity of the relevance degree is extracted; and the filter non-association is designed. The information is reasonably filtered; all data are processed, converted to the same interval for processing; data protocol is implemented; and data effective information feature mining is implemented based on the SVM algorithm. The simulation results show that the algorithm is effective for filtering big data and has high precision and superior performance, which shows good application value.

The TPCx-BigBench (TPCx-BB) is a TPC Express benchmark, which is designed to measure the performance of big data analytics systems. It contains 30 use cases that simulate big data processing, big data storage, big data analytics, and reporting. We have used this benchmark to evaluate the performance of software and hardware components for big data systems. It has very good coverage on different data types and provides enough scalability to address data size and node scaling problems. We have gained lots of meaningful insights through this benchmark to design analytic systems. In the meantime, we also found we cannot merely rely on TPCx-BB to evaluate and design an end-to-end big data systems. There are some gaps between an analytics system and a real end-to-end system. The whole data flow of a real end-to-end system should include data ingestion, which moves data from where it is originated into a system where it can be stored and analyzed such as Hadoop. Data ingestion may be challenging for businesses at a reasonable speed in order to maintain a competitive advantage. However, TPCx-BB cannot help on performance evaluation of software and hardware for data ingestion. Big data is composed of three dimensions: Volume, Variety, and Velocity. The Velocity refers to the high speed in data processing: real-time or near real-time. With big data technology widely used, real-time and near real-time processing become more popular. There is very strict limitation on bandwidth and latency for real-time processing. TPCx-BB cannot help on performance evaluation of software and hardware for real-time processing. This paper mainly discusses these experiences and lessons in practice using TPCx-BB. Then, we provide some advices to extend TPCx-BB to cover data ingestion and real-time processing. We also share some ideas how to implement TPCx-BB coverage.

Nowadays, big data has become a hot research topic. It gives fresh impetus to the economic and social development. However, the huge value of big data also makes it the focus of attacks. Big data security incidents occur frequently in recent years. The security supervision capacities for big data do not match its important role. Data provenance which describes the origins of data and the process by which it arrived the current state, is an effective approach for data supervision. For the full use of provenance in big data supervision, a provenance model which defines the concepts used to represent the provenance types and relations is required to be built in advance, but current provenance models do not adapt to big data scenarios well. In this paper, we comprehensively consider the characteristics of big data and the requirements of data security supervision, extend the widely used provenance model PROV-DM by subtyping and new relation definition, and propose a big data provenance model (BDPM) for data supervision. BDPM model supports the provenance representation of various data types and diverse data processing modes to represent the entire data transformation process through different components in the big data system, and defines new relations to enrich provenance analysis functions. Based on BDPM model, we introduce the constraints that should be satisfied in the construction of valid provenance graph and present the data security supervision methods via provenance graph analysis. Finally, we evaluated the satisfiability of BDPM model through a case study.

Big data systems handle vast volumes of structured and unstructured information using distributed computing frameworks like Apache Hadoop, Spark, and Flink. These platforms rely heavily on computational resources, and CPU utilization becomes a critical factor in determining the performance and efficiency of data processing operations. Effective CPU usage directly impacts throughput, latency, and the ability to meet service-level objectives. In large-scale clusters, workloads vary from CPU-intensive tasks such as real-time analytics, data mining, and machine learning to more I/O-driven operations like data ingestion or storage management. Poor CPU distribution can lead to overloaded nodes while others remain underutilized, ultimately reducing system efficiency. Traditional scheduling mechanisms in big data platforms are often rule-based and may not respond well to dynamic or unpredictable workloads, which contributes to suboptimal CPU usage. When CPU utilization remains consistently high, it can trigger job slowdowns, task failures, and increased power consumption, negatively affecting overall performance. Conversely, low CPU utilization indicates underused resources, leading to waste in computational capacity and energy. Additionally, container technologies and control groups (cgroups) support fine-tuned CPU allocation and isolation, ensuring fair usage across concurrent users or tasks in multi-tenant environments. In more advanced implementations, machine learning algorithms are used to forecast CPU needs and schedule jobs more efficiently by learning from historical patterns and usage behavior. Reinforcement learning approaches have also shown potential in achieving balanced CPU usage by adapting policies based on feedback and environment changes. These intelligent mechanisms allow big data systems to optimize CPU allocation continuously, adapting to fluctuating demand without manual intervention. Efficient CPU management contributes significantly to faster job execution, reduced operational costs, and higher cluster reliability. As data volumes and complexity continue to grow, maintaining optimal CPU utilization remains a priority for organizations aiming to derive timely insights while maximizing infrastructure value. Basic CPU utilization approaches are exhibiting performance limitations. This paper addresses performance limitations related to CPU utilization.

The big data concept has been explosive, revealing, and transformative across manufacturing industries as it provides deeper insights into manufacturing operations for decision-making. However, big green data (BGA), a dedicated subset of big data, is not adequately structured for comprehensive sustainability analysis, particularly in smart factories. With our proposed green data balance (GDB), there will be accountability for each input and output composition in a production unit within the production value chain (PVC). Data will be exhaustively and accurately collected in each workshop to help uncover unknown issues in a production value chain while facilitating the development of sustainability metrics or index systems. Additionally, a structured big green data system will fuel “greentelligence”, using intelligent systems and technologies to speed up digitalization toward sustainable manufacturing by measuring, tracking, and minimizing adverse environmental impacts. Lastly, with the support of the cognitive intelligence data analytic system (CIDAS), real-time and near real-time comprehensive sustainability analytics can be performed, leading to Self-X metacognitive adjustments and corrective actions.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

"Big data" has become a major force of innovation across enterprises of all sizes. New platforms with increasingly more features for managing big datasets are being announced almost on a weekly basis. Yet, there is currently a lack of any means of comparability among such platforms. While the performance of traditional database systems is well understood and measured by long-established institutions such as the Transaction Processing Performance Council (TCP), there is neither a clear definition of the performance of big data systems nor a generally agreed upon metric for comparing these systems. In this article, we describe a community-based effort for defining a big data benchmark. Over the past year, a Big Data Benchmarking Community has become established in order to fill this void. The effort focuses on defining an end-to-end application-layer benchmark for measuring the performance of big data applications, with the ability to easily adapt the benchmark specification to evolving challenges in the big data space. This article describes the efforts that have been undertaken thus far toward the definition of a BigData Top100 List. While highlighting the major technical as well as organizational challenges, through this article, we also solicit community input into this process.

Big Data analytics has become an invaluable tool in a wide variety of businesses for exploiting the wealth of Big Data that they now have access to. As a result, various solutions within different categories of Big Data systems are emerging to meet their needs. In this paper we use the TPC-H benchmark to compare the performance of four Big Data systems picked from the major categories of Big Data platforms: a commercial parallel relational database (from the traditional DBMS world), Hive and Spark SQL (from the SQL-on-Hadoop world), and AsterixDB (from the world of NoSQL systems). All of these systems have sufficiently rich query APIs and runtime systems to run TPC-H in its full form. On the other hand, the systems also have major differences in terms of their architectures, preferred storage formats, support for complex schema definitions, and approaches to query processing. This makes them a very interesting set of representative Big Data systems to compare. We present the results that we obtained through running these systems at different TPC-H scales using various settings, and we analyze a selected set of interesting query results in more detail to explore the trade-offs between performance, storage formats, and schema definitions. A follow-up discussion is included as well to summarize the lessons learned from this effort.