An Optimized Straggler Mitigation Framework for Large-Scale Distributed Computing Systems

The article describes the development of an architecture for cloud storage of melons based on a Residue Number System (RNS). The developed architecture is evaluated in terms of the reliability of the stored data, and its comparative analysis is carried out with existing data storage methods. The architecture of a multi-cloud big data storage and processing system based on the principles of modular arithmetic is proposed. The proposed modification of the cloud storage and data processing system allows increasing the reliability and fault tolerance of the stored and processed data. Single-level and two-level models for reliable storage of big data were developed based on the simulation results. The comparative analysis showed the advantage of models based on modular arithmetic over models based on traditional arithmetic.

In today's world, the increase in the volume and variety of data used in the process of military management leads to an increase in the requirements for their storage and processing. To effectively manage military operations, it is necessary to have information about various aspects of military activity, such as the location of military units, the availability of resources, the assessment of potential threats, and much more. Modern military operations require the processing and analysis of a large amount of data generated in real time. Data collected from sensors, satellites, probes, sensors, and other sources can be vast in volume and diverse in structure. The increase in the volume and heterogeneity of data used in the process of military management leads to an increase in the requirements for their storage and processing. This presents military organizations with the task of creating effective and reliable data storage and processing systems. The article considers the possibilities of using «Big Data» technology, which allow collecting, storing, processing and analyzing large volumes of various data in real time. The use of modern data management technologies can help increase the efficiency of military activities and ensure fast and accurate decision-making at various levels of management. This can be achieved through the use of artificial intelligence, machine learning and data analysis technologies that can help military structures perform calculations and forecasts, optimize the use of resources and increase the efficiency of solving various tasks. Also, an important component of the effective use of «Big Data» technologies in the military sphere is ensuring the protection of data from unauthorized access, as well as ensuring the reliability and stability of data storage and processing systems in the face of possible cyberattacks. The article analyzes the Hadoop system, which can be used to store, process and analyze «Big Data» in the military sphere.

In the big data processing systems, the amount of data is increasing. At the same time, the real-time requirement of data processing and analysis is higher and higher. Therefore, it is required that the big data processing and analysis systems have better performance. Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. However, job scheduling is a difficult NP-hard problem. There are many factors that need to be considered for job scheduling. For example, jobs have dependencies among stages, therefore we should not allocate resources to tasks that are not ready. Sometimes, there are constraints between jobs. These are a challenge to the scheduling performance of big data processing and analysis systems. In this paper, we try to solve the problem by translating it into Boolean Satisfiability Problem (SAT) which is an exact method. SAT-based scheduling algorithm is not a new approach, but in the past it mainly used to solve the static scheduling problems. For dynamic scheduling system, it requires all problems to be solved within a limited time, which is a challenge for SAT encoding. In this paper, we refer to the previous SAT solution to the Job Shop Scheduling Problem, and adjust the algorithm to meet the requirements of the big data processing system. At the same time, we optimized the coding approach and reduced the number of clauses. Thus, the efficiency of the problem solved is improved to meet the performance requirements. The experimental results show that the number of clauses is reduced by more than 30%, and the processing time of the SAT solver to get the solution can be reduced by more than 50%. To demonstrate its effectiveness, we have also implemented our new job scheduler in Apache Hadoop YARN, and validated its effectiveness

Many Big Data processing applications nowadays run on large-scale multi-tenant clusters. Due to hardware heterogeneity and resource contentions, straggler problem has become the norm rather than the exception in such clusters. To handle the straggler problem, speculative execution has emerged as one of the most widely used straggler mitigation techniques. Although a number of speculative execution mechanisms have been proposed, as we have observed from real-world traces, the questions of when and where to launch speculative copies have not been fully discussed and hence cause inefficiencies on the performance and energy of Big Data applications. In this paper, we propose a performance model and an energy consumption model to reveal the performance and energy variations with different speculative execution solutions. We further propose a window-based dynamic resource reservation and a heterogeneity-aware copy allocation technique to answer the when and where questions for speculative executions. Evaluations using real-world traces show that our proposed technique can improve the performance of Big Data applications by up to 30% and reduce the overall energy consumption by up to 34%.

This experience report describes the successful application of statistical software testing to a large-scale realworld equipment replacement project. The Data Processing System at Dungeness B nuclear power station is obsolescent and needs to be replaced by a new system involving different technology. The new Data Processing and Control System (DPCS) involves a large amount of software code, with the potential for software failures, and therefore confidence needs to be established in the dependability of the new software-based system. Statistical software testing (ST) was used as one technique to do this. The aim was to support the claim that with high confidence the higher integrity parts of the replacement software have a failure probability of better than 10-2 failures per demand (fpd). This paper describes how this has been achieved by careful design of the statistical tests and by the development of a large number of supporting tools and procedures. A total of 395 statistical tests with no failure was achieved providing at least 98% confidence that the safety claim has been met.

Many social networking sites such as Facebook and Twitter have been used for sharing knowledge and information among social entities. Social entities in these social networks are often linked by some interdependency such as friendship or following relationships. Amounts of high volumes of high-value data can be easily collected and generated in these social networking sites. As the size of the social network keeps increasing in the current era of big data, there are many real-life situations in which a social entity wants to find those frequently followed groups of social entities from these big data so that he can follow the same groups. In this paper, we present a big data mining algorithm to discover following patterns from these big social network data. Evaluation results show the efficiency and practicality of our algorithm in big social network mining for the following patterns.

Addressing the policy challenges and opportunities of “Big data”

In the fourth industrial revolution, various inter-mediation such as theories, techniques, or implementation would be used. It contains computational intelligence, applied soft computing and fuzzy logic and artificial neural networks, intelligent contents security, model driven architecture and meta-modeling, multimedia contents processing and retrieval, vehicular N/W, big data, intelligence information processing, convergence/complex contents, smart learning, intelligent contents design management, methodology and design theory, intelligent media contents convergence/complex media, social media and collective intelligence, and social media big data analytic. To consider these interesting and significant issues which have related to the fast developing area of smart media and application, the editors have edited this special issue Smart Media and Application, and finally, it selected eleven papers. In Motion data acquisition method for motion analysis in golf by Hwang et al,1 it studied the process a system that obtains the same motion as the actual one by acquiring information on the motion of the subject using 15 inertial sensors, and using the information on the actual joint length and initial direction of the subject extracted from a depth camera. As a result, it shows that measurement error of information on the joint length and the foot stance of subjects extracted through the depth camera ranges from a minimum of 4.4% to a maximum of 6.94%. In NSCT domain–based secure multiple watermarking technique through lightweight encryption for medical images, which is written by Thakur et al,2 it defined non-subsampled contourlet transform (NSCT) domain, in which the proposed scheme first partitions the host image into sub-components and then calculates the entropy values. In Environmental monitoring system for intelligent stations by Li et al,3 it proposed the design of the high-speed rail station environmental monitoring system based on the LoRa communication technology, and it aims to enable the real-time monitoring of temperature, humidity, illuminance, and noise decibels in high-speed railway stations. Efficient dummy generation for considering obstacles and protecting user location is by Song et al.4 This paper was to study an efficient dummy creating techniques to improve user privacy protection, and as a result, it showed that it improved on other recent techniques. Key node selection based on a genetic algorithm for fast patching in social networks by Kim et al5 provided considerable amounts of personal information and shares this information with friends without space-time limitations. It tried to improve the patch propagation speed; it was important to select key nodes that are the starting points of the patch process. In this paper, the authors proposed a key node selection scheme based on a genetic algorithm to find the most significant contribution nodes of the patch propagation. Following the simulation result, it showed the proposed scheme propagates patches more rapidly than the existing one. In Effective computer-assisted pronunciation training based on phone-sensitive word recommendation by Jo et al,6 they proposed a computer-assisted pronunciation training system targeting unacceptable pronunciation due to confusion among contextual allophones, a problem that often emerges from phoneme-based feedback provided during pronunciation training. Finally, it showed that the proposed system results in the improvement of pronunciation skills through training sessions that use recommended words containing phoneme pairs that were initially pronounced incorrectly. In Dataset retrieval system based on automation of data preparation with dataset description model by Mun et al,7 it tried to overcome the problem by proposing a dataset description model that can express the requirements for data processing and dataset retrieval system based on automated data preparation and it could possibly provide good quality datasets for statistical learning applications using data preparation methods such as data acquisition, refinement, and organization. In An enhanced 3DCNN-ConvLSTM for spatiotemporal multimedia data analysis by Wang et al,8 it used 3DCNN in CNN part and ConvLSTM in RNN part. In A web-based group decision support system for multicriteria problems written by Conceição et al,9 it used a multiagent system to combine and process this information, using virtual agents that represent each decision-marker. The high level of usability that the system provides will contribute to an easier acceptance and adoption of this kind of systems. In Intelligent and semantic threshold schemes for security in cloud computing by Ogiela and Snasel,10 it tried to enhanced with new classes of solutions, including semantic protocols of secret sharing, and it is subjected to the process of splitting and hiding that could take various forms including that of services. In Estimation of a physical activity energy expenditure with a patch-type sensor module using artificial neural network by Kang et al,11 it proposed the most accurate method using a wireless patch-type sensor to predict the energy expenditure of physical activities. Through the optimization of the prediction of energy expenditure of physical activities using the neural network algorithm, it achieved RMSE of 0.1893, R2 of 0.91 for the energy expenditures of aerobic and anaerobic exercises. We would like to thank you our sincere appreciation of the valuable contributions to all authors.

The global database research community has greatly impacted the functionality and performance of data storage and processing systems along the dimensions that define data, i.e., volume, velocity, variety, and veracity. Locally, over the past five years, we have also been working on varying fronts. Among our contributions are: (1) establishing a vision for a database-inspired big data analytics system, which unifies the best of database and distributed systems technologies, and augments it with concepts drawn from compilers (e.g., iterations) and data stream processing, as well as (2) forming a community of researchers and institutions to create the Stratosphere platform to realize our vision. One major result from these activities was Apache Flink, an open-source big data analytics platform and its thriving global community of developers and production users. Although much progress has been made, when looking at the overall big data stack, a major challenge for database research community still remains. That is, how to maintain the ease-of-use despite the increasing heterogeneity and complexity of data analytics, involving specialized engines for various aspects of an end-to-end data analytics pipeline, including, among others, graph-based, linear algebra-based, and relational-based algorithms, and the underlying, increasingly heterogeneous hardware and computing infrastructure. At TU Berlin, DFKI, and the Berlin Big Data Center (BBDC), we aim to advance research in this field via the Mosaics project. Our goal is to remedy some of the heterogeneity challenges that hamper developer productivity and limit the use of data science technologies to just the privileged few, who are coveted experts.

Earth observation (EO) data are remote sensing image data obtained by air-borne or space-borne sensors, which has the characters of distribution and heterogeneity. Because these data are owned by different space agencies. With Petabyte-scale data volumes of Earth Observation (EO) obtained, the EO data has been one of the most important part of big data. The characters of distribution and heterogeneity of big EO data storage and processing systems owned by different space agencies brings challenges to integrate and use these data. To meet the big EO data challenges, the new generation Spatial data infrastructure is needed to provide the solution and implementation technique to big EO data storage, access, process and analysis. The concept and architecture of Next-generation national EO Spatial Data Infrastructure (EO-SDI) is proposed in the paper. Key techniques are introduced to solve the complexity of distributed and heterogeneous EO data integration and on-demand processing. An interoperable national EO data system of systems is developed to integrate distributed EO archives of eight EO data centers.

The worldwide spread of social networks provides new opportunities to collect data for various applied research. The geotagging function, which is present in most social networks, is extremely useful in the case of spatial research. For urban researchers, this is more than just an opportunity to collect a huge amount of information about the behavior of people in space, but also to analyze the space itself. However, this type of data has both significant advantages and limitations. The article includes highlighting the most significant characteristics that urban researchers need to consider working with social networks: their volume (the ability to work with them as big data), non-reactivity (they are reproduced by the users themselves) and additional interpretations (they allow us to understand the meanings of certain elements of space for individual individuals or groups). The authors also illustrate the research opportunities provided by online data in combination with geotagging with various empirical examples.

In the field of education and training, digital transformation supports education and training innovation in the direction of reducing lectures to developing learner capacity, increasing self-learning ability, creating learning opportunities anytime, anywhere, personalizing learning, contributing to creating a learning society and lifelong learning. The explosion of IoT technology platforms, Big Data, AI, SMAC (social networks - mobile - big data analysis - cloud computing) is forming digital education infrastructure. Accordingly, many smart education models are being developed on the basis of information technology application, making access to a huge knowledge base in the network environment fast and easy... Maximum support for personalized learning (each person learns a curriculum and a unique learning method. This is done automatically by the information technology system); help the interaction between families, schools, teachers and students take place quickly and promptly

The term Big Data can be defined as analysis a large volume of data in unstructured databases, organizations of different segment and size, has been employing the Big Data Analytics philosophy as support tool strategic to anticipate valuable insights and trends on the behavior of consumers and their expectations, thus gaining a competitive advantage in the market in which they operate. However, to extract information from values as the goal of turning that volume of data into predictive information or insights is still a big challenge in Big Data. The main objective of this work is to present an implementation of a Big Data project, using data originated from social networks, as well as text mining techniques and machine learning, through the implementation of K-Means and SVM algorithms, with intention of identifying patterns from dengue outbreaks, through analyzes that show insights of probable outbreaks of dengue in a particular region of Brazil. The results obtained indicate that the implemented project had a satisfactory yield if compared of the data collected of the Ministry of Health of Brazil, thus indicating a potential for utilization of its purpose. It is observed that the main advantage of the analyzes in Big Data is related to the possibility of use the unstructured data that can be obtained in social networks, e-commerce sites, among others and structured data obtained from traditional databases and from this union, to extract information from values that can be used to benefit of organizations, thus allowing know the future behaviors and thereby act in a preventive way.

The ASTRI Mini-Array is an international project led by the Italian National Institute for Astrophysics (INAF) to build and operate an array of nine 4-m class Imaging Atmospheric Cherenkov Telescopes (IACTs) at the Observatorio del Teide (Tenerife, Spain). The system is designed to perform deep observations of the galactic and extragalactic gamma-ray sky in the TeV and multi-TeV energy band, with important synergies with other ground-based gamma-ray facilities in the Northern Hemisphere and space-borne telescopes. As part of the overall software system, the ASTRI (Astrofisica con Specchi a Tecnologia Replicante Italiana) Team is developing dedicated systems for Data Processing, Simulation, and Archive to achieve effective handling, dissemination, and scientific exploitation of the ASTRI Mini-Array data. Thanks to the high-speed network connection available between Canary Islands and Italy, data acquired on-site will be delivered to the ASTRI Data Center in Rome immediately after acquisition. The raw data will be then reduced and analyzed by the Data Processing System up to the generation of the final scientific products. Detailed Monte Carlo simulated data will be produced by the Simulation System and exploited in several data processing steps in order to achieve precise reconstruction of the physical characteristics of the detected gamma rays and to reject the overwhelming background due to charged cosmic rays. The data access at different user levels and for different use cases, each one with a customized data organization, will be provided by the Archive System. In this contribution we present these three ASTRI Mini-Array software systems, focusing on their main functionalities, components, and interfaces.

Mining big graph data is an important problem in the graph mining research area. Although cloud computing is effective at solving traditional algorithm problems, mining frequent patterns of a massive graph with cloud computing still faces the three challenges: 1) the graph partition problem, 2) asymmetry of information, and 3) pattern-preservation merging. Therefore, this paper presents a new approach, the cloud-based SpiderMine (c-SpiderMine), which exploits cloud computing to process the mining of large patterns on big graph data. The proposed method addresses the above issues for implementing a big graph data mining algorithm in the cloud. We conduct the experiments with three real data sets, and the experimental results demonstrate that c-SpiderMine can significantly reduce execution time with high scalability in dealing with big data in the cloud.