Distributed Graph Path Queries Using Spark

Fault-tolerance has long been a feature of database systems, with transactions supporting the structuring of applications so as to ensure continuation of updating applications in spite of machine failures. For read-only queries the perceived wisdom has been that support for fault-tolerance is too expensive to be worthwhile. Distributed query processing is coming to be seen as a promising way of implementing applications that combine structured data and analysis operations in dynamic distributed settings such as computational grids. Such a query may be long-running and having to redo the whole query after a failure may cause problems (e.g. if the result may trigger business or safety critical activities). This work describes and evaluates a new scheme for adding fault-tolerance to distributed query processing through a rollback-recovery mechanism. The high level expression of user requests in a physical algebra offers opportunities for tuning the fault-tolerance provision so as to reduce the cost, and give better performance than employment of generic fault-tolerance mechanisms at the lowest level of query processing. This paper outlines how the publicly-available OGSA-DQP computational grid-based distributed query processing system can be modified to include support for fault-tolerance and presents a performance evaluation which includes measurements of the cost of both protocol overheads and rollback-recovery, for a set of example distributed queries.

In distributed query processing systems where caching infrastructure is distributed and scales with the number of servers, it is becoming more important to orchestrate and leverage a large number of cached objects in distributed caching systems seamlessly as the present trend is to build large scalable distributed systems by connecting small heterogeneous machines. With a large scale distributed caching system, a scheduling policy must consider both cache hit ratio and system load balance to optimize multiple queries. A scheduling policy that considers system load but not cache hit ratio often fails to reuse cached data by not assigning a query to the sever that has data objects the query needs. On the contrary, a scheduling policy that considers cache hit ratio but not system load balance may suffer from system load imbalance. To maximize the overall system throughput and to reduce query response time, a multiple query scheduling policy must balance system load and also leverage cached objects. In this paper, we present a distributed query processing framework that exhibits high cache hit ratio while achieving good system load balance. In order to seamlessly manage our distributed scalable caching system, our framework performs autonomic cached data migrations to improve cache hit ratio. Our experiments show that our proposed query scheduling policy and data migration policy significantly improve system throughput by achieving high cache hit ratio while avoiding system load imbalance.

Query Processing is a key determinant in the overall performance of distributed databases. It requires processing of data at their respective sites and transmission of the same between them. These together constitute a distributed query processing strategy (DQP). DQP aims to arrive at an efficient query processing strategy for a given query. This strategy involves generation of efficient query plans for a distributed query. In case of distributed relational queries, the number of possible query plans grows exponentially with an increase in the number of relations accessed by the query. This number increases further when the relations, accessed by the query, have replicas at different sites. Such a large search space renders it infeasible to find optimal query plans. This paper presents a query plan generation algorithm that attempts to generate optimal query plans, for a given query, using genetic algorithm. The query plans so generated involve fewer sites, thus leading to efficient query processing. Further, experimental results show that the proposed algorithm converges quickly towards optimal query plans for an observed crossover and mutation probability.

Multiple query scheduling for distributed semantic caches

In the environment of wide-area networks such as the Internet, distributed query processing is becoming problematic due to changes of the underlying networks and user requirements. In this new context, conventional query processing strategies with the homogeneous assumption will not work well, because they are unable to adapt to unexpected changes in the performance of the communication networks. In this paper, we address the issue of how to make distributed query processing aware of these changes. We introduce the idea of integrating distributed query processing with quality of service (QoS) management and accordingly illustrate our view of QoS-aware distributed query processing.

The rapid expansion of the Internet of Things (IoT) and Big Data (BD) has led to security challenges. Securing IoT-BD against cyberattacks is necessary. An increasing number of applications are being implemented on BD platforms due to the rapid proliferation of data on the Internet. As the volume of data increases, the possibility of intrusions on the platform correspondingly increases. Conventional Intrusion Detection Systems (IDS) are ineffective for managing the extensive volume of historical data and unable to fulfil the security demands of BD platforms. This research aims to propose a novel intrusion detection model using Binary Sand Cat Swarm Optimization and Spatiotemporal Transformer Neural Network (BSCSO-STNN) model to address these issues. The CIC-IoT-23 and Bot-IoT datasets are collected and applied to train the model for evaluation. The developed BSCSO-STNN model is deployed in an Apache Spark (APS) framework. The datasets are initially preprocessed in this framework with data cleaning, oversampling, label encoding, and normalization. After preprocessing, the data is applied to the BSCSO for feature selection. Using the selected features, the STNN model performs binary and multiclass classification for both datasets. The BSCSO-STNN model attained 99.08% accuracy, 98.78% detection rate, 99.02% precision, and 98.94% F1-score using the CIC-IoT-23 dataset. The model attained 99.04% accuracy, 98.81% detection rate, 98.97% precision, and 98.95% F1-score for the BoT-IoT dataset in multiclass classification. The developed model outperformed all the current models in this research and demonstrated its accuracy in detecting intrusions.

Query processing in Web search engines today is mainly performed within a single site or data center, which is required to scale as the Web grows and users require fast answers to their queries. Constraints in the size and cost of data centers, however, may limit the scalability of search engines. Multi-site search engines that perform distributed query processing represent one way to overcome such constraints. Each site processes locally as many queries as possible, keeping latency low without contacting remote sites. Forwarding a query to remote sites depends on the document collection of remote sites. Multi-site search engines pose several new challenges. When a site updates its index, it has to inform other sites. The updates, however, are not instantaneous due to the volume of data exchanged or possible network failures. During the period of time that there are index inconsistencies across sites, queries may not be forwarded optimally. In this work, we investigate the impact of index inconsistencies on a distributed query processing algorithm, when there are index updates, and we observe that delayed index information propagation reduces the effectiveness of query processing, because queries are less likely to be routed optimally.

Actionability is a mode of revealing actionable knowledge in the form of Action Rules from large datasets. Action rule imparts in the form of recommendations as how a data object can change from one value to another more desirable value. The towering production of data in the recent years, due to increased usage of web, social media and IoT, has led to the age of big data. Also, abundant usage of cloud storages and cloud based services causes the data to be spread around the globe. This requires more time and space for a single computer to cope with such widespread data. Ecosystems like Hadoop MapReduce, Spark have been introduced to store, process and retrieve back the data efficiently in a distributed fashion. Data mining finds substantial improvements over such distributed frameworks to process huge volume of data and acquire knowledge from them in a short span of time. In this paper, we present an approach SARGS: Specific Action Rule discovery based on Grabbing Strategy, to build more specific Action Rules using Apache Spark framework and evaluate the results with our previous Hadoop MapReduce system (MR-Random Forest Algorithm for Distributed Action Rules Discovery). Also, we propose a novel approach to distribute data in a distributed environment to get more optimal Action Rules and upgraded ARoGS algorithm to get more specific Action Rules.

Anomaly detection is a major issue for several applications such as industrial failure detection, cybersecurity or transport. Several approaches, such as statistical methods, machine learning and sketch, have been explored by different research communities to detect anomalies in an increasingly challenging context. Indeed, facing the huge volume of data generated at an increasingly fast speed, the response time of the algorithms and their distributivity have become determining criteria, in addition to their accuracy in detecting anomalies. We focus in this paper on the unsupervised anomaly detection algorithm based on binary trees: Isolation Forest. It is a very powerful algorithm with an excellent accuracy and a very low execution time thanks to its linear complexity. In particular, we study the architecture of two distribution solutions of Isolation Forest based on the Apache Spark framework. We then compare the performance of these two solutions by testing them against 4 real commonly used datasets.

Big data tends to concentrate on the data volume and variety which requires large cluster capabilities to process diverse and heterogeneous data. Currently, NoSQL/Hadoop-based cluster frameworks are known to excel at handling this form of data by scaling across nodes and distributed query processing. But for certain data sizes, relational databases can also support these workloads. In this paper, we support this claim over a popular relational database engine, Microsoft* SQL Server* 2019 (pre-release candidate) using a big data benchmark, BigBench. Our work in this paper is the industry first case study that runs BigBench on a single node environment powered by Intel® XeonTM processor 8164 product family and enterprise-class Intel® SSDs. We make the following two contributions: (1) present response times of all 30 BigBench queries when run sequentially to showcase the advanced analytics and machine learning capabilities integrated within SQL Server 2019, and (2) present results from data scalability experiments over two scale factors (1 TB, 3 TB) to understand the impact of increase in data size on query runtimes. We further characterize a subset of queries to understand their resource consumption requirements (CPU/IO/memory) on a single node system. We will conclude by correlating our initial engineering study to similar research studies on cluster-based configurations providing a further hint to the potential of relational databases to run reasonably scaled big-data workloads.

Introduction/BackgroundTo determine the clinical characteristics predictive of optimal cytoreduction among women with epithelial ovarian malignancy who underwent cytoreductive surgery in a tertiary government hospital.MethodologyThis retrospective cohort study identified 218 patients...

The issue of data quality has emerged as a critical concern, as low-quality data can impede data sharing, diminish intrinsic value, and result in economic losses. Current research on data quality assessment primarily focuses on four dimensions: intrinsic, contextual, presentational, and accessibility quality, with intrinsic and presentational quality mainly centered on data content, and contextual quality reflecting data usage scenarios. However, existing approaches lack consideration for the behavior of data within specific application scenarios, which encompasses the degree of participation and support of data within a given scenario, offering valuable insights for optimizing resource deployment and business processes. In response, this paper proposes a data contribution assessment method based on maximal sequential patterns of behavior paradigms (DecentralDC). DecentralDC is composed of three steps: (1) mining the maximal sequential patterns of sharing and exchange behavior paradigms; (2) determining the weights of these paradigms; (3) calculating the contribution of sharing and exchange databases combined with data volume. To validate our approach, two sharing and exchange scenarios of different scales are established. The experimental results in two scenarios validate the effectiveness of our method and demonstrate a significant reduction in cumulative regret and regret rate in data pricing due to the introduction of data contribution. Specifically, compared to the most competitive baseline, the improvements of mean average precision in two scenarios are 6% and 8%. The code and simulation scenarios have been open-sourced and are available at https://github.com/seukgcode/DecentralDC.

With the advent of the Legacy Survey of Space and Time, time-domain astronomy will be faced with an unprecedented volume and rate of data. Real-time processing of variables and transients detected by such large-scale surveys is critical to identifying the more unusual events and allocating scarce follow-up resources efficiently. We develop an algorithm to identify these novel events within a given population of variable sources. We determine the distributions of magnitude changes (dm) over time intervals (dt) for a given passband f, , and use these distributions to compute the likelihood of a test source being consistent with the population or being an outlier. We demonstrate our algorithm by applying it to the DECam multiband time-series data of more than 2000 variable stars identified by Saha et al. in the Galactic Bulge that are largely dominated by long-period variables and pulsating stars. Our algorithm discovers 18 outlier sources in the sample, including a microlensing event, a dwarf nova, and two chromospherically active RS CVn stars, as well as sources in the blue horizontal branch region of the color–magnitude diagram without any known counterparts. We compare the performance of our algorithm for novelty detection with the multivariate Kernel Density Estimator and Isolation Forest on the simulated PLAsTiCC data set. We find that our algorithm yields comparable results despite its simplicity. Our method provides an efficient way for flagging the most unusual events in a real-time alert-broker system.

The lead-zinc sintering process has the characters of complex mechanism, multi-parameter and strong interconnected coupling. This paper presents an intelligent modeling and optimization method based on comprehensive production Indices. First, the neural network prediction model of the comprehensive production indices is proposed, which is synthesizing some techniques, including gray correlation analysis, principal components analysis, and neural network and so on. And the target function is deduced and the model based on comprehensive production Indices is given by using the multi-objective optimization technique. At last, this paper proposes an improved multi-objective particle swarm optimization algorithm based on SPEA2 to calculate the optimization executing parameters. The applied results of actual runs show that the intelligent modeling and optimization method proposed in this paper attains a better industrial effect, and provides an effective and new way to implement the global optimization.

An automated hybrid approach using clustering and nature inspired optimization technique for improved tumor and tissue segmentation in magnetic resonance brain images