Datalog and emerging applications

Recursive query processing has experienced a recent resurgence, as a result of its use in many modern application domains, including data integration, graph analytics, security, program analysis, networking and decision making. Due to the large volumes of data being processed, several research efforts across multiple communities have explored how to scale up recursive queries, typically expressed in Datalog. Our experience with these tools indicate that their performance does not translate across domains---e.g., a tool designed for large-scale graph analytics does not exhibit the same performance on program-analysis tasks, and vice versa. Starting from the above observation, we make the following two contributions. First, we perform a detailed experimental evaluation comparing a number of state-of-the-art Datalog systems on a wide spectrum of graph analytics and program-analysis tasks, and summarize the pros and cons of existing techniques. Second, we design and implement our own general-purpose Datalog engine, called RecStep, on top of a parallel single-node relational system. We outline the techniques we applied on RecStep, as well as the contribution of each technique to the overall performance. Using RecStep as a baseline, we demonstrate that it generally out-performs state-of-the-art parallel Datalog engines on complex and large-scale Datalog evaluation, by a 4-6X margin. An additional insight from our work is that it is possible to build a high-performance Datalog system on top of a relational engine, an idea that has been dismissed in past work.

We present a unified declarative platform for specifying, implementing, and analyzing secure networked information systems. Our work builds upon techniques from logic-based trust management systems, declarative networking, and data analysis via provenance. We make the following contributions. First, we propose the secure network datalog (SeNDlog) language that unifies Binder, a logic-based language for access control in distributed systems, and Network Datalog, a distributed recursive query language for declarative networks. SeNDlog enables network routing, information systems, and their security policies to be specified and implemented within a common declarative framework. Second, we extend existing distributed recursive query processing techniques to execute SeNDlog programs that incorporate authenticated communication among untrusted nodes. Third, we demonstrate that distributed network provenance can be supported naturally within our declarative framework for network security analysis and diagnostics. Finally, using a local cluster and the PlanetLab testbed, we perform a detailed performance study of a variety of secure networked systems implemented using our platform.

Big data has emerged as the next technological revolution in IT industry after cloud computing and the Internet of Things. With the development of climate observing systems, particularly satellite meteorological observation and high-resolution climate models, and the rapid growth in the volume of climate data, climate prediction is now entering the era of big data. The application of big data will provide new ideas and methods for the continuous development of climate prediction. The rapid integration, cloud storage, cloud computing, and full-sample analysis of massive climate data makes it possible to understand climate states and their evolution more objectively, thus predicting the future climate more accurately. This paper describes the application status of big data in operational climate prediction in China; it analyzes the key big data technologies, discusses the future development of climate prediction operations from the perspective of big data, speculates on the prospects for applying climatic big data in cloud computing and data assimilation, and puts forward the notion of big data-based super-ensemble climate prediction methods and computerbased deep learning climate prediction methods.

Hybrid transactional and analytical processing (HTAP) systems like SAP HANA make it much simpler to manage both operational load and analytical queries without ETL, separate data warehouses, et al. To represent both transactional and analytical business logic in a single database system, stored procedures are often used to express analytical queries using control flow logic and DMLs. Optimizing these complex procedures requires a fair knowledge of imperative programming languages as well as the declarative query language. Therefore, unified optimization techniques considering both program and query optimization techniques are essential for achieving optimal query performance. In this paper, we propose a novel unified optimization technique for efficient iterative query processing. We present a notion of query motion that allows the movement of SQL queries in and out of a loop. Additionally, we exploit a new cost model that measures the quality of the execution plan with consideration for queries and loop iterations. We describe our experimental evaluation that demonstrates the benefit of our technique using both a standard decision support benchmark and real-world workloads. An extensive evaluation shows that our unified optimization technique enumerates plans that achieve performance improvements of up to an order of magnitude faster than plans generated by the existing loop-invariant code motion technique.

BackgroundLong non-coding RNAs (lncRNAs) play vital roles in many important biological processes in plants. Currently, a large fraction of plant lncRNA studies center at lncRNA identification and functional analysis. Only a few plant lncRNA studies focus on understanding their evolutionary history, which is crucial for an in-depth understanding of lncRNAs. Therefore, the integration of large volumes of plant lncRNA data is required to deeply investigate the evolution of lncRNAs.ResultsWe present a large-scale evolutionary analysis of lncRNAs in 25 flowering plants. In total, we identified 199,796 high-confidence lncRNAs through data integration analysis, and grouped them into 5497 lncRNA orthologous families. Then, we divided the lncRNAs into groups based on the degree of sequence conservation, and quantified the various characteristics of 756 conserved Arabidopsis thaliana lncRNAs. We found that compared with non-conserved lncRNAs, conserved lncRNAs might have more exons, longer sequence length, higher expression levels, and lower tissue specificities. Functional annotation based on the A. thaliana coding-lncRNA gene co-expression network suggested potential functions of conserved lncRNAs including autophagy, locomotion, and cell cycle. Enrichment analysis revealed that the functions of conserved lncRNAs were closely related to the growth and development of the tissues in which they were specifically expressed.ConclusionsComprehensive integration of large-scale lncRNA data and construction of a phylogenetic tree with orthologous lncRNA families from 25 flowering plants was used to provide an oversight of the evolutionary history of plant lncRNAs including origin, conservation, and orthologous relationships. Further analysis revealed a differential characteristic profile for conserved lncRNAs in A. thaliana when compared with non-conserved lncRNAs. We also examined tissue specific expression and the potential functional roles of conserved lncRNAs. The results presented here will further our understanding of plant lncRNA evolution, and provide the basis for further in-depth studies of their functions.

The estimation and control of resource usage is now an important challenge in an increasing number of computing systems. In particular, requirements on timing and energy arise in a wide variety of applications such as internet of things, cloud computing, health, transportation, and robots. At the same time, parallel computing, with (heterogeneous) multi-core platforms in particular, has become the dominant paradigm in computer architecture. Predicting resource usage on such platforms poses a difficult challenge. Most work on static resource analysis has focused on sequential programs, and relatively little progress has been made on the analysis of parallel programs, or more specifically on parallel logic programs. We propose a novel, general, and flexible framework for setting up cost equations/relations which can be instantiated for performing resource usage analysis of parallel logic programs for a wide range of resources, platforms, and execution models. The analysis estimates both lower and upper bounds on the resource usage of a parallel program (without executing it) as functions on input data sizes. In addition, it also infers other meaningful information to better exploit and assess the potential and actual parallelism of a system. We develop a method for solving cost relations involving the max function that arise in the analysis of parallel programs. Finally, we instantiate our general framework for the analysis of logic programs with Independent And-Parallelism, report on an implementation within the CiaoPP system, and provide some experimental results. To our knowledge, this is the first approach to the cost analysis of parallel logic programs.

The reliability of quality management in industrial processes mainly depends on information about the traceability, precision and accuracy of a measurement system as well as on its systematic bias. The progressive development of networking and sensing in industrial machinery facilitates a quality-related process monitoring regarding information of measurement systems and singular sensor nodes. Hereby, integration on the information level is mandatory. Furthermore, information from the shop floor and from enterprise applications is needed to provide a consistent and integrated quality analysis. Thereby, these systems use different standards and technologies for exportation and propagation of data. Besides, integrative quality management and data analysis require enriched data that does not only comprise, for example, the measured value and its standard-dependent unit on the sensor level; rather, additional information is needed (e.g. the production process or the time and place of measurement). In this paper, a framework is presented that facilitates the semantic integration and analysis of measurement and enterprise data according to real-time requirements. Semantic technologies are used to encode the meaning of the data from the application code. Herewith, the data is automatically annotated using terms and concepts taken from the application domain. Furthermore, a semantic integration and transformation process is facilitated. Thus, subsequent integration and, most importantly, analysis processes can take advantage of these terms and concepts using specialized analysis algorithms. Besides, the conceptual application of the presented framework and processes in a high-pressure-die-casting scenario is presented.

The Qatar Computing Research Institute (QCRI), a member of Qatar Foundation for Education, Science and Community Development, started its activities in early 2011. QCRI is focusing on tackling large-scale computing challenges that address national priorities for growth and development and that have global impact in computing research. QCRI has currently five research groups working on different aspects of computing, these are: Arabic Language Technologies, Social Computing, Scientific Computing, Cloud Computing, and Data Analytics. The data analytics group at QCRI, DA@QCRI for short, has embarked in an ambitious endeavour to become a premiere world-class research group by tackling diverse research topics related to data quality, data integration, information extraction, scientific data management, and data mining. In the short time since its birth, DA@QCRI has grown to now have eight permanent scientists, two software engineers and around ten interns and postdocs at any given time. The group contributions are starting to appear in top venues.

Due to the ubiquity of spatial data applications and the large amounts of spatial data that these applications generate and process, there is a pressing need for scalable spatial query processing. In this paper, we present new techniques for spatial query processing and optimization in an in-memory and distributed setup to address scalability. More specifically, we introduce new techniques for handling query skew that commonly happens in practice, and minimizes communication costs accordingly. We propose a distributed query scheduler that uses a new cost model to minimize the cost of spatial query processing. The scheduler generates query execution plans that minimize the effect of query skew. The query scheduler utilizes new spatial indexing techniques based on bitmap filters to forward queries to the appropriate local nodes. Each local computation node is responsible for optimizing and selecting its best local query execution plan based on the indexes and the nature of the spatial queries in that node. All the proposed spatial query processing and optimization techniques are prototyped inside Spark, a distributed memory-based computation system. Our prototype system is termed LocationSpark. The experimental study is based on real datasets and demonstrates that LocationSpark can enhance distributed spatial query processing by up to an order of magnitude over existing in-memory and distributed spatial systems.

Data integration from multiple autonomous data sources has emerged as an important practical problem. The key requirement for such data integration is that owners of such data need to cooperate in a competitive landscape in most of the cases. The research challenge in developing a query processing solution is that the answers to the queries need to be provided while preserving the privacy of the data sources. In general, allowing unrestricted read access to the whole data may give rise to potential vulnerabilities as well as may have legal implications. Therefore, there is a need for privacy preserving database operations for querying data residing at different parties. In this paper, we propose a new query processing technique using third parties in a peer-to-peer system. We propose and evaluate two different protocols for various database operations. Our scheme is able to answer queries without revealing any useful information to the data sources or to the third parties. Analytical comparison of the proposed approach with other recent proposals for privacy-preserving data integration establishes the superiority of the proposed approach in terms of query response time

Big data and cloud computing (BDCloud) collectively offer a paradigm shift in the way businesses are now acquiring, using and managing information technology. With the fast growth of this paradigm, we argue that each and every CS and IT students should be equipped with foundation knowledge in this collective paradigm and should possess hand-on-experiences in managing big data applications in clouds to acquire skills that are necessary to meet current and future industry demands. This poster presents our research that proposes gradual and systematic integration of big data and cloud computing related topics into multiple core (required) courses of CS/IT curriculum. The poster, supported by a NSF grant, will be useful for CS/IT students and their instructors as it identifies big data and cloud computing related topics that are important to cover, finds a sequence of the prescribed topics that can be incorporated into existing core courses most effectively, and suggests specific core courses in which their coverage might find an appropriate context. The poster further identifies the major challenges this proposed intervention may encounter and provides a deeper analysis of them. Finally, the poster describes our experience of implementing one such course with proposed interventions during Fall of 2016 semester. The pre- post- test results that measure student opinion and understanding of big data and cloud computing topics are presented in the poster and demonstrate improved student interest and learning.

Due to the importance of linear algebra and matrix operations in data analytics, there is significant interest in using relational query optimization and processing techniques for evaluating (sparse) linear algebra programs. In particular, in recent years close connections have been established between linear algebra programs and relational algebra that allow transferring optimization techniques of the latter to the former. In this paper, we ask ourselves which linear algebra programs in MATLANG correspond to the free-connex and q-hierarchical fragments of conjunctive first-order logic. Both fragments have desirable query processing properties: free-connex conjunctive queries support constant-delay enumeration after a linear-time preprocessing phase, and q-hierarchical conjunctive queries further allow constant-time updates. By characterizing the corresponding fragments of MATLANG, we hence identify the fragments of linear algebra programs that one can evaluate with constant-delay enumeration after linear-time preprocessing and with constant-time updates. To derive our results, we improve and generalize previous correspondences between MATLANG and relational algebra evaluated over semiring-annotated relations. In addition, we identify properties on semirings that allow to generalize the complexity bounds for free-connex and q-hierarchical conjunctive queries from Boolean annotations to general semirings.

Chapter 4 - Optimization techniques and computational intelligence with emerging trends in cloud computing and Internet of Things

Big Data and Cloud Computing has become trends and technologies of the day. The daily explosion of data means that it's better to have big data included in the applications. Whereas cloud computing is allowing users to use platforms according to their time, convenience and affordability. Cloud computing seems to be a perfect vehicle for hosting big data workloads. However, working on big data in the cloud brings its many challenges of reconciling two contradictory design principles. Cloud computing is based on the concepts of consolidation and resource pooling, but big data systems (such as Hadoop) are built on the shared nothing principle, where each and every node is independent and self-sufficient. The integration of big data with cloud, businesses and educational institution can have a better direction to the near future. Various analytics and technology involved in coupling of big data with cloud computing, the challenges involved in this process, trends applications of the domain and security issues involved have been discussed in this paper.

20 years after writing Enterprise Application Integration, data integration is still an afterthought when it comes to the cloud deployments I see. Enterprises moving to cloud tend to focus on the move itself, and not as much on what they need when they get there. While this may be a common practice, it's not a best practice. Data integration is needed because we just re-hosted some of our data on a remote cloud service. Thus, the inventory system that's still running on a mainframe in the data center needs to share data with the sales order system that's now on AWS. In other words, our data integration problem domain just got bigger and more complex.