Simplifying the Structural Recursion of the Data Funnel Interface

Chapter 3 - Basic Table Creation and Data Manipulation

We present a logic, called fuzzy attribute logic (FAL), for reasoning about formulas describing particular attribute dependencies. The formulas are of a form A ⇒ B where A and B are collections of attributes. Our formulas can be interpreted in two ways. First, in data tables with entries containing degrees to which objects (table rows) have attributes (table columns). Second, in database tables where each domain is equipped with a similarity relation assigning a degree of similarity to any pair of domain elements. We assume that the scale of degrees is equipped with fuzzy logical connectives and forms an arbitrary complete residuated lattice. This covers many structures used in fuzzy logic applications as well as structures used in formal systems of fuzzy logic. If the scale contains only two degrees, 0 (falsity) and 1 (truth), two well-known calculi become particular cases of our logic. Namely, with the first interpretation, our logic coincides with attribute logic used in formal concept analysis; with the second interpretation, our logic coincides with Armstrong system for reasoning about functional dependencies. We prove completeness of FAL over arbitrary complete residuated lattices in two versions. First, in the ordinary style, completeness asserts that a formula A ⇒ B is entailed by a collection T of formulas if A ⇒ B is provable from T. Second, in the graded style, completeness asserts that a degree to which A ⇒ B is entailed by a collection T of formulas equals a degree to which A ⇒ B is provable from T.

Column scan, or predicate evaluation and filtering over a column of data in a database table, is an important primitive for data mining and data warehousing. In this paper, we present our study on accelerating column scan using a massively parallel accelerator. With a design that takes full advantage of the characteristics of the memory hierarchy and parallel execution in such processors, we have achieved very attractive speedup performance that significantly exceeds previously reported results, making the use of such an accelerator for this type of primitives much more viable. Running on a general purpose graphic processor unit (GPGPU), NVidia GTX 280 GPU, the GPU version is about 5-6 times faster than an implementation on an eight-core CPU, or over 40 times faster than that on a single-core CPU.

To secure cloud storage and enforce access control, data encryption has become essential, given the ever increasing cyber threat everywhere. Attribute-based Encryption (ABE) crypto systems are widely considered as a promising solution under such a context for its security strength, scalability and control flexibility. One major challenge, however, for applying ABE-based techniques in real world applications is its high overhead in various aspects. In this research, we are particularly concerned with the storage size expansion in existing ABE schemes. This combined with the vast-size nature of the cloud data poses an enormous challenge to the effective usage of the cloud data storage space and affects the utility of data deduplication. Normally, data deduplication is carried out based on identifying similar and even identical contents both within and between data files, however, these patterns will be destroyed after performing data encryption using any semantically secure encryption scheme including ABE. In this research, we focus on ciphertexts deduplication under ABE, which to our best knowledge is the first of such an effort. Our fundamental observation stems from the structure of ABE ciphertexts and the possible similarities among different access structures. We show how to design a secure ciphertext deduplication scheme based on a classical CP-ABE scheme by innovatively modifying the construction with a recursive algorithm, eliminating the duplicated secrets and adding additional randomness to some certain ciphertext. We then give a detailed analysis on the proposed scheme with respect to both efficiency and security. To thoroughly assess the performance of the proposed scheme, we also implement a prototype system and conduct comprehensive experiments, which shows that our ciphertext reduplication scheme could reduce up to 80% computation and storage cost in the best case.

Aiming at the problems of long time consuming and low accuracy in traditional methods of abnormal data detection in power measurement automation system, this paper studies the methods of abnormal data detection in power measurement automation system. Design the data storage structure table of the electric power metering automation system database, and repair the missing data and denoise the data in the data table. Perform PAA calculation on the data to get the data feature sequence. After the P clustering algorithm pre-clusters the data, the iForest model is used to detect abnormal data to complete the research on the method. The experimental results show that the proposed detection method has the advantages of short detection time and high precision of 91.26–95.67%.KeywordsPower measurement automation systemAbnormal dataDetection methodIforest

BackgroundThe Cancer Genome Atlas Project (TCGA) is a National Cancer Institute effort to profile at least 500 cases of 20 different tumor types using genomic platforms and to make these data, both raw and processed, available to all researchers. TCGA data are currently over 1.2 Petabyte in size and include whole genome sequence (WGS), whole exome sequence, methylation, RNA expression, proteomic, and clinical datasets. Publicly accessible TCGA data are released through public portals, but many challenges exist in navigating and using data obtained from these sites. We developed TCGA Expedition to support the research community focused on computational methods for cancer research. Data obtained, versioned, and archived using TCGA Expedition supports command line access at high-performance computing facilities as well as some functionality with third party tools. For a subset of TCGA data collected at University of Pittsburgh, we also re-associate TCGA data with de-identified data from the electronic health records. Here we describe the software as well as the architecture of our repository, methods for loading of TCGA data to multiple platforms, and security and regulatory controls that conform to federal best practices.ResultsTCGA Expedition software consists of a set of scripts written in Bash, Python and Java that download, extract, harmonize, version and store all TCGA data and metadata. The software generates a versioned, participant- and sample-centered, local TCGA data directory with metadata structures that directly reference the local data files as well as the original data files. The software supports flexible searches of the data via a web portal, user-centric data tracking tools, and data provenance tools. Using this software, we created a collaborative repository, the Pittsburgh Genome Resource Repository (PGRR) that enabled investigators at our institution to work with all TCGA data formats, and to interrogate these data with analysis pipelines, and associated tools. WGS data are especially challenging for individual investigators to use, due to issues with downloading, storage, and processing; having locally accessible WGS BAM files has proven invaluable.ConclusionOur open-source, freely available TCGA Expedition software can be used to create a local collaborative infrastructure for acquiring, managing, and analyzing TCGA data and other large public datasets.

Large experiments and high-performance computer models generate many petabytes of data. While Cloud Computing systems may meet the needs for analyzing these petabytes by harnessing the computing power of many distributed computers, the key challenge in effectively utilizing such a distributed system is the data management process, including storage, indexing, searching, accessing, and transferring data. Most analysis tasks perform computations on a subset of a large data records satisfying some user specified constraints on attribute (variable) values. This subsetting procedure is extremely important in that it reduces the network traffic to and from the cloud facilities. However, selected data records often span many different data files, and extracting the values out these files can be time-consuming especially if the number of files is large. This work addresses this challenge of working with a large number of files. We use a set of astronomical data set as an example and use an efficient database indexing technique, called FastBit, to significantly speed up the subsetting and thus optimize network usage. Overall, we aim to provide transparent and highly efficient attribute-based data access to scientists through a web-based Astronomy Data Analysis Portal. We will discuss the system design, and options for managing an extremely large number of files while minimizing network usage and latency.

To solve financing difficulty in rural SMEs and provide fast and efficient financing service for SMEs, this paper applies internet technology to design and develop an informationalized service platform. By repeated communication and exchange with bank and rural SMEs, this paper performs system requirement analysis in detail, clarifies the functions to be realized in the system and completes module design in system as well as database structure design. The system realization adopts ASP.NET to rapidly construct Web system under internet environment, applies jQuery, ADO.NET technology to realize data exchange between foreground and background, website and database and finally uses SQL Server 2008 to set up database and data table. Finally, it completes overall test between corresponding module and system. Results show that feasibility and effectiveness of platform reach the design requirement.

To test bridging techniques for the harmonized estimation of forest biodiversity indicators for each of the selected essential features a common database was constructed and populated with raw NFI data provided by some of the COST Action E43 participating countries. The database was structured with five tables in a relational database: one table for descriptive plot data, one for tree level data, one for deadwood pieces, one for shrub data and one for ground vegetation. The database was populated with data for 320,023 trees, data for 25,639 pieces of deadwood, 12,588 shrub records and 34,364 ground vegetation records from 14,638 NFI plots provided by 13 European countries and the USA.

The accelerator physics group has put in place a computer information service derived in the main from the lattice design data stored in the CCD Sybase relational database management system (rDBMS). For example, the lattice described in the database "rhic92r0p3" corresponds to revision 0.3 of the RHIC 92 lattice, as tracked through CCB protocols. These primary data describing the beam related aspects of the machine which, in principle, we are building are stored in database tables tuned to the needs of the beamline designers and the codes that they use. Translation to the needs of users in other parts of the RHIC project is not immediately evident, although the data are one of the prime sources driving the machine construction. Accordingly, we wish to provide data and access mechanisms suited to the different parts of the project. We have started by building data files and database tables containing surveying information, CAD drawing instructions, beam parameters and so on. Our aim is to provide access to these data and to guarantee that they reflect the best knowledge of the detailed machine definition at any given time - a definition that is reasonable to base construction decisions on. Because what data and tools are available will be changing we wish to encourage computer access rather than paper communication. For those who have no desire to become computer gurus, we will try to provide advice and utilities.

This dataset includes all the data files that were used for the studies in my PhD dissertation Together and apart: Perfective verbs with a prefix and the semelfactive suffix -nu- in Contemporary Standard (2019). Most of the files involve data tables with annotated corpus data from the Russian National Corpus (www.ruscorpora.ru). Every tabular data file is available in two formats: .csv and .xlsx. The data files are numbered so that they are shown in the same order as they become relevant in the dissertation, and their contents are described in the ReadMe files with the same number. In addition, the dataset contains the R script and .txt file that was used in order to make the plot in Chapter 2.

The Landsat 7 Processing System (LPS) processes Landsat 7 Enhanced Thematic Mapper Plus (ETM-+) instrument data into large, contiguous segments called "subintervals." The LPS-processed subinterval products must be subsetted and reformatted before the Level 1 processing systems can ingest them. The initial full subintervals produced by the LPS are stored mainly in HDF Earth Observing System (HDF-EOS) format, an extension to the Hierarchical Data Format (HDF). The final LOR products are stored in native HDF. The HDF and HDF-EOS application programming interfaces (APIs) can be used for extensive data subsetting and data reorganization. How the HDF and HDF-EOS APIs may be used to efficiently subset, format, and organize Landsat 7 LOR data to create various configurations of LOR products is discussed.

The multiple-trait derivative-free REML set of programs was written to handle partially missing data for multiple-trait analyses as well as single-trait models. Standard errors of genetic parameters were reported for univariate models and for multiple-trait analyses only when all traits were measured on animals with records. In addition to estimating (co)variance components for multiple-trait models with partially missing data, this paper shows how the multiple-trait derivative-free REML set of programs can also estimate SE by augmenting the data file when not all animals have all traits measured. Although the standard practice has been to eliminate records with partially missing data, that practice uses only a subset of the available data. In some situations, the elimination of partial records can result in elimination of all the records, such as one trait measured in one environment and a second trait measured in a different environment. An alternative approach requiring minor modifications of the original data and model was developed that provides estimates of the SE using an augmented data set that gives the same residual log likelihood as the original data for multiple-trait analyses when not all traits are measured. Because the same residual vector is used for the original data and the augmented data, the resulting REML estimators along with their sampling properties are identical for the original and augmented data, so that SE for estimates of genetic parameters can be calculated.

Constraints on Data Sharing

This article describes the design and implementation of (OR/SM), a computerized modeling environment based on Structured Modeling. The uniqueness of OR/SM is in the following: (1) the use of oracle Tools and Database as the delivery platform; (2) automatic and interactive links to sas, a powerful and widely used commercial statistical analysis software system and optimization solver; and (3) an interactive link to qs (Quantitative Systems)—a commercial software package for solving a wide range of operations management models. Some other key features are: (1) automatic generation of relational database tables for model data; (2) interactive checking of model syntax and semantics; and (3) automatic generation of several reference documents. Examples from blending, inventory control, and marketing mix management are used to illustrate the capabilities of OR/SM.