Transforming spreadsheet data types using examples

The emergence of the digital era has led to growth in various types of data in a cloud. In fact, there may be three fourth of the total data will be treated as big data. In many organisations, massive volume of both structured and unstructured data sit idle. Various categories of data are complex for pre-processing, analysing, storing and visualising. Cloud computing provides suitable platform for big data analytics for the storage and for predicting customer behaviour to sell products. Unstructured data like emails, notes, messages, documents, notifications and Twitter comments (including from IoT devices) remains untapped and is not stored in a relational database. Valuable information on pricing, customer behaviour and competitors may be inhumed within unstructured data. This makes cloud-based analytics as an effective research field to address several issues and risks need to be reduced. So we propose a method to extract and cluster sentiment information from various types of unstructured text data from social networks by using SVM classifiers combined with lexicons and machine learning for sentiment analysis of customer behaviour feedback. The method has performed efficient data collection, data loading and efficiently performs sentiment analysis on deep and hidden web.

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

Fusion 2+ of air-to-air engagement involves pressing, real-time constraints and very large amounts of imperfect data. Real-time data acquired during an air-to-air engagement will have different types of imperfection; two representative classes of imperfection are vagueness and ambiguity in the data. However, the current approaches of managing Fusion 2+ are limited to utilize either vague data or ambiguous data. The most popular fusion technique for vague data is Fuzzy Logic, and for ambiguous data, the Bayesian Network. The challenge addressed in this proposal is to explore the framework of a hybrid processing Fusion 2+ model that can formally process both vague (fuzzy) and ambiguous (probabilistic) data types. There are two major issues for building this Fusion 2+ model. The first issue is to mathematically integrate the heterogeneous models, which have different domains, probability and possibility. The second issue is to programmatically integrate two different S/Ws. For solving the first issue, this research explores and adopts two novel transformation methods between probability and possibility and compares the sensitivity between methods. Also this research provides an Object Oriented Tool for building a hybrid model by adopting an Application Programming Interface, so that we can model the complex (multi-to-multi) Fusion 2+ model of an air-to-air engagement.

Security of Big Data is a huge concern. In a broad sense, Big Data contains two types of data: structured and unstructured. Providing security to unstructured data is more difficult than providing security to structured data. In this paper, we have developed an approach to provide adequate security to unstructured data by considering types of data and their sensitivity levels. We have reviewed the different analytics methods of Big Data to build nodes of different types of data. Each type of data has been classified to provide adequate security and enhance the overhead of the security system. To provide security to a data node, and a security suite has been designed by incorporating different security algorithms. Those security algorithms collectively form a security suite which has been interfaced with the data node. Information on data sensitivity has been collected through a survey. We have shown through several experiments on multiple computer systems with varied configurations that data classification with respect to sensitivity levels enhances the performance of the system. The experimental results show how and in what amount the designed security suite reduces overhead and increases security simultaneously.

1.4 - Demographics of Corporate Data

Synthesis from examples enables non-expert users to generate programs by specifying examples of their behavior. A domain-specific form of such synthesis has been recently deployed in a widely used spreadsheet software product. In this paper we contribute to foundations of such techniques and present a complete algorithm for synthesis of a class of recursive functions defined by structural recursion over a given algebraic data type definition. The functions we consider map an algebraic data type to a string; they are useful for, e.g., pretty printing and serialization of programs and data. We formalize our problem as learning deterministic sequential top-down tree-to-string transducers with a single state (1STS). The first problem we consider is learning a tree-to-string transducer from any set of input/output examples provided by the user. We show that, given a set of input/output examples, checking whether there exists a 1STS consistent with these examples is NP-complete in general. In contrast, the problem can be solved in polynomial time under a (practically useful) closure condition that each subtree of a tree in the input/output example set is also part of the input/output examples. Because coming up with relevant input/output examples may be difficult for the user while creating hard constraint problems for the synthesizer, we also study a more automated active learning scenario in which the algorithm chooses the inputs for which the user provides the outputs. Our algorithm asks a worst-case linear number of queries as a function of the size of the algebraic data type definition to determine a unique transducer. To construct our algorithms we present two new results on formal languages. First, we define a class of word equations, called sequential word equations, for which we prove that satisfiability can be solved in deterministic polynomial time. This is in contrast to the general word equations for which the best known complexity upper bound is in linear space. Second, we close a long-standing open problem about the asymptotic size of test sets for context-free languages. A test set of a language of words L is a subset T of L such that any two word homomorphisms equivalent on T are also equivalent on L. We prove that it is possible to build test sets of cubic size for context-free languages, matching for the first time the lower bound found 20 years ago.

Recently, various detection approaches that identify anomalous events (e.g., discoloration, contamination) by analyzing data collected from smart meters (so-called structured data) have been developed for many water distribution systems (WDSs). However, although some of them have showed promising results, meters often fail to collect/transmit the data (i.e., missing data) thus meaning that these methods may frequently not work for anomaly identification. Thus, the clear next step is to combine structured data with another type of data, unstructured data, that has no structural format (e.g., textual content, images, and colors) and can often be expressed through various social media platforms. However, no previous work has been carried out in this regard. This study proposes a framework that combines structured and unstructured data to identify WDS water quality events by collecting turbidity data (structured data) and text data uploaded to social networking services (SNSs) (unstructured data). In the proposed framework, water quality events are identified by applying data-driven detection tools for the structured data and cosine similarity for the unstructured data. The results indicate that structured data-driven tools successfully detect accidents with large magnitudes but fail to detect small failures. When the proposed framework is used, those undetected accidents are successfully identified. Thus, combining structured and unstructured data is necessary to maximize WDS water quality event detection.

A huge amount of healthcare data (structured and unstructured) is currently available to medical specialists, indicating details of clinical symptoms. Each type of data provides information that must be properly analyzed for healthcare diagnosis. To simplify the diagnostic process, avoid misdiagnosis as well as early detection, artificial intelligence (AI) that aims to mimic human cognitive functions may be employed. Current AI techniques that are used for structured data include machine learning methods, such as the classical support vector machine, artificial neural network, and the modern deep learning. Natural language processing is mainly used for unstructured data. In this paper, we have adopted artificial neural network by using adaptive learning algorithms to handle diverse types of cardiovascular clinical data and integrate them into categorized major cardiovascular disease outputs such as heart failure, aortic aneurysm, cardiomyopathy, cerebrovascular disease, etc. These outputs are then shared as reports to patients as well as doctors by an efficient report sharing scheme called APDRChain, which combines blockchain and structured peer-to-peer network techniques with clever cryptography to create a consensus mechanism. The evaluation results show that APDRChain can achieve higher efficiency and satisfy the security requirements in report sharing.

Software developers have long been supported by a variety of tools, such as version control systems (e.g., GIT), issue tracking systems (e.g., BugZilla), and mailing list services (e.g., Mailman). These tools accumulate a wide range of information that is recorded in the repositories these tools store their data in. This information is comprised of two significantly different types of data: structured and unstructured data. Structured data (e.g., source code or execution traces) has a well-established structure and grammar, thus is straightforward to parse and use with computer machinery. Unstructured data (e.g., documentation, discussions, comments, or customer support requests) consists of a mixture of natural language text, snippets of structured data, and noise. Mining unstructured data is very challenging since out-of-the box approaches adopted from related fields, such as Natural Language Processing and Information Retrieval, cannot be directly applied in software engineering. To tackle challenges faced when mining unstructured data and make the knowledge contained in unstructured data repositories accessible to both practitioners and researchers, we organize the 2nd workshop on Mining Unstructured Data (MUD'12). The aim is to provide a unique interactive venue for discussing in-depth challenges, approaches, and applications and share experiences, and results on the topic of mining software unstructured data.

Big data is a valuable asset for organisation as it analyses and help to understand the customers, changes within their business environment, market analysis and future trends. The big data is multifaceted (different data types and versatile), and mostly exists in unstructured formats. The extraction of value from this data is challenging. The usability and productivity of this multifaceted unstructured data is greatly compromised. A number of factors and associated reasons affect the usability of unstructured big data. The present research work investigates these factors and associated reasons behind the usability issues of multifaceted unstructured big data. The identification of these factors contribute to develop solutions to reduce the lack of usability of highly unstructured big data. A detailed study of existing literature followed by survey questionnaire has been conducted to identify the factors and their reasons. Descriptive statistics has been used to analyse and interpret the data and results.

Big Data is used to store huge volume of both structured and unstructured data which is so large and is hard to process using current / traditional database tools and software technologies. The goal of Big Data Storage Management is to ensure a high level of data quality and availability for business intellect and big data analytics applications. Graph database which is not most popular NoSQL database compare to relational database yet but it is a most powerful NoSQL database which can handle large volume of data in very efficient way. It is very difficult to manage large volume of data using traditional technology. Data retrieval time may be more as per database size gets increase. As solution of that NoSQL databases are available. This paper describe what is big data storage management, dimensions of big data, types of data, what is structured and unstructured data, what is NoSQL database, types of NoSQL database, basic structure of graph database, advantages, disadvantages and application area and comparison of various graph database.

Visualization has become a valuable means for data exploration and analysis. Interactive visualization combines expressive graphical representations and effective user interaction. Although interaction is an important component of visualization approaches, much of the visualization literature tends to pay more attention to the graphical representation than to interaction. The goal of this work is to strengthen the interaction side of visualization. Based on a brief review of general aspects of interaction, we develop an interaction-oriented view on visualization. This view comprises five key aspects: the data, the tasks, the technology, the human, as well as the implementation. Picking up these aspects individually, we elaborate several interaction methods for visualization. We introduce a multi-threading architecture for efficient interactive exploration. We present interaction techniques for different types of data e.g., multivariate data, spatio-temporal data, graphs) and different visualization tasks (e.g., exploratory navigation, visual comparison, visual editing). With respect to technology, we illustrate approaches that utilize modern interaction modalities (e.g., touch, tangibles, proxemics) as well as classic ones. While the human is important throughout this work, we also consider automatic methods to assist the interactive part. In addition to solutions for individual problems, a major contribution of this work is the overarching view of interaction in visualization as a whole. This includes a critical discussion of interaction, the identification of links between the key aspects of interaction, and the formulation of research topics for future work with a focus on interaction.

Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.

Clinical risk prediction models powered by electronic health records (EHRs) are becoming increasingly widespread in clinical practice. With suicide-related mortality rates rising in recent years, it is becoming increasingly urgent to understand, predict, and prevent suicidal behavior. Here, we compare the predictive value of structured and unstructured EHR data for predicting suicide risk. We find that Naive Bayes Classifier (NBC) and Random Forest (RF) models trained on structured EHR data perform better than those based on unstructured EHR data. An NBC model trained on both structured and unstructured data yields similar performance (AUC = 0.743) to an NBC model trained on structured data alone (0.742, p = 0.668), while an RF model trained on both data types yields significantly better results (AUC = 0.903) than an RF model trained on structured data alone (0.887, p < 0.001), likely due to the RF model’s ability to capture interactions between the two data types. To investigate these interactions, we propose and implement a general framework for identifying specific structured-unstructured feature pairs whose interactions differ between case and non-case cohorts, and thus have the potential to improve predictive performance and increase understanding of clinical risk. We find that such feature pairs tend to capture heterogeneous pairs of general concepts, rather than homogeneous pairs of specific concepts. These findings and this framework can be used to improve current and future EHR-based clinical modeling efforts.

Calculation of Protein Structures with Ambiguous Distance Restraints. Automated Assignment of Ambiguous NOE Crosspeaks and Disulphide Connectivities