Structured and Unstructured Big Data Analytics

Big data is a valuable asset for organisation as it analyses and help to understand the customers, changes within their business environment, market analysis and future trends. The big data is multifaceted (different data types and versatile), and mostly exists in unstructured formats. The extraction of value from this data is challenging. The usability and productivity of this multifaceted unstructured data is greatly compromised. A number of factors and associated reasons affect the usability of unstructured big data. The present research work investigates these factors and associated reasons behind the usability issues of multifaceted unstructured big data. The identification of these factors contribute to develop solutions to reduce the lack of usability of highly unstructured big data. A detailed study of existing literature followed by survey questionnaire has been conducted to identify the factors and their reasons. Descriptive statistics has been used to analyse and interpret the data and results.

The existing big data platforms take data through distributed processing platforms and store them in a data lake. The architectures such as Lambda and Kappa address the real-time and batch processing of data. Such systems provide real time analytics on the raw data and delayed analytics on the curated data. The data denormalization, creation and maintenance of a columnar dimensional data warehouse is usually time consuming with no or limited support for unstructured data. The system introduced in this paper automatically creates and dynamically maintains its data warehouse as a part of its big data pipeline in addition to its data lake. It creates its data warehouse on structured, semi-structured and unstructured data. It uses Machine Learning to identify and create dimensions. It also establishes relations among data from different data sources and creates the corresponding dimensions. It dynamically optimizes the dimensions based on the crowd sourced data provided by end users and also based on query analysis.

The term “Big Data” refers to a large volume of information usually in terabytes and petabytes. It includes both structured and unstructured data. Unstructured data is conventionally text-heavy, but may also contain data such as facts, dates and numbers. To use this unstructured information effectively, it needs to be processed and organized. Taxonomy is considered as a powerful way of organizing the information. For automatic taxonomy generation, various techniques have been proposed in the past. However, the substantial nature of big data presently crosses the processing abilities of traditional techniques. Thus, to meet this challenge an extensible and scalable technique is required to potentially accelerate the process of taxonomy generation and its evolution upon arrival of new data, hence catering large amount of unstructured big data. This paper proposes a technique for both the taxonomy generation and evolution on Apache Spark infrastructure. The proposed technique is evaluated on a text dataset of a computing domain. The evaluation results show that the technique presented in this paper outperformed the existing techniques in terms of time and quality metrics. The time and quality-based evaluation showed that the use of MapReduce environment has resolved the scalability issues of current taxonomy generation and evolution process.

Geospatial data in databases have shifted to conform to the characteristics of big-data in tandem with the development of the Internet, mobile Internet, cloud computing, and especially, spatial data acquisition technologies. Faced with tackling spatial big data, traditional spatial database management techniques based on Relational Database Management Systems have encountered problems including the unstructured characteristics of the spatial object, the high scalability of storage capacity, and the high concurrency in big data application environment. This paper focuses on the mainstream of NoSQL databases that successfully deal with unstructured big data and are widely used in Internet applications, but lack of spatial characteristics. The data operational and query modes cannot meet the requirments of GIS applications. To resolve this problem, this paper proposes a strategy that takes a NoSQL database as a warehouse for spatial big data and a traditional spatial database as the application server. The storage system architecture and the key technology and solutions are discussed. A prototype system was developed based on MongoDB, PostgreSQL and SQLite to verify the feasibility and effectiveness of the strategy.

The significance of big data is increasingly acknowledged across all sectors, including medicine. Moreover, the trend of data trading is on the rise, particularly in exchanging other data for medical data to rejuvenate the medical industry. This study aimed to discern the facilitating factors of healthcare data trade. We assessed five medical data market platforms on October, 2022, based on three criteria: (a) clarity in articulating the data for sale; (b) transparency in specifying the data costs; and (c) explicit indication that payment grants data access. This helped identify the traded medical data types. Additionally, we anonymously surveyed 43 representatives from medical device companies about their demand for medical data trading, achieving a response rate of 66%. Of the medical data traded on these platforms, 93.34% was structured, while 5.66% was unstructured, indicating an imbalance. Although there was a higher demand for structured medical data, there was also interest in purchasing unstructured medical data. Unstructured big data are crucial for medical device development, fueling the demand for trading such data. Many stakeholders view the data market as essential and are willing to procure medical data. Consequently, medical device companies will need methods to acquire unstructured medical data for developing innovative and enhanced medical devices.

The emergence of the digital era has led to growth in various types of data in a cloud. In fact, there may be three fourth of the total data will be treated as big data. In many organisations, massive volume of both structured and unstructured data sit idle. Various categories of data are complex for pre-processing, analysing, storing and visualising. Cloud computing provides suitable platform for big data analytics for the storage and for predicting customer behaviour to sell products. Unstructured data like emails, notes, messages, documents, notifications and Twitter comments (including from IoT devices) remains untapped and is not stored in a relational database. Valuable information on pricing, customer behaviour and competitors may be inhumed within unstructured data. This makes cloud-based analytics as an effective research field to address several issues and risks need to be reduced. So we propose a method to extract and cluster sentiment information from various types of unstructured text data from social networks by using SVM classifiers combined with lexicons and machine learning for sentiment analysis of customer behaviour feedback. The method has performed efficient data collection, data loading and efficiently performs sentiment analysis on deep and hidden web.

With the rapid development of Big Data and the necessity for analyzing their huge volumes, the issue of Unstructured Data analysis in social media was appeared. The Data analysis process is very important in all fields as to make decisions at the right time and over certain facts. The usage of social media has become the latest trend in today's world in which users send, read posts known as `message' and communicate with various groups. Users are sharing their regular life, posting their views on everything like products and locations. This data is extremely unstructured, making it hard to analyze. Machine learning technology offers important data preparation techniques for processing large-scale data to extract knowledge, e.g., classifying data. Extract useful information from social media data is essential to success in the big data age. Therefore, fresh strategies are needed for handling huge quantities of unstructured data and finding the hidden information in these data and achieving better data analysis outcomes, In this paper, the proposed framework recommends the construction of a machine-learning model capable of analyzing unstructured text data with highly accuracy compared to other machine learning algorithms.

The amount of data in our world has been exploding. Hence, integrating, managing and analyzing large amounts of data – i.e. big data - will become a key issue for businesses for better operating and competing in today’s markets. We strongly believe that big data are really useful if used in a smart way. So we introduce the concept of Smart Data that is web and enterprise structured and unstructured big data with explicit and implicit semantics that leverages context to understand intent for better driving business processes and for better and more informed decisions making. This paper proposes a language able to give a representation of big data based on ontologies and a system that implements an approach capable to satisfy the increasing need for efficiency and scalability in semantic data management. The proposed MANTRA Language allows for: (i) representing the semantics of data by knowledge representation constructs; (ii) acquiring big data from disparate heterogeneous sources (e.g. data bases, documents), integrating and managing them; (iii) reasoning and querying with big data. The syntax of the proposed language is partially derived from logic programming, but the semantic is completely revised. The novelty of the language we propose is that a class can be thought of as a flexible collection of structurally heterogeneous individuals that have different properties (schema-less). The language also allows executing efficient querying and reasoning for revealing implicit knowledge. These have been achieved by using a triple-based data persistency model and a scalable No-SQL storage system.

Employee Satisfaction and Corporate Performance in the UK

Technological discoveries and the spread of digital technologies require social scientists to revise basic philosophical ideas about the world, society, and man. Issues related to the preservation of freedom, protection of private life and privacy of a person are becoming especially relevant in the digital society. The article considers big data as a tool of social control, supervision and as a means of predicting the behavior of individuals. With the help of big data, large platform companies can shape the core values, cultural codes and consumer behavior of users. The evolution of models of social control is considered: panopticon, synopticon, superpanopticon. While the panopticon is characterized by disciplinary power, the digital society that uses unstructured big data is characterized by instrumental power. The theory of "supervisory capitalism" by S. Zuboff is analyzed and two features of "supervisory capitalism" are distinguished: profit extraction and forecasting of user behavior. In conclusion, having considered the main models of social surveillance, the author highlights the features of social control in a digital society. Social control becomes more subtle, latent, total and indifferent towards a person. The superpanopticon is developing as a social model associated with universal observation based on big data.

Topics such as computational sources and cloud-based transmission and security of big data have turned out to be a major new domain of exploration due to the exponential evolution of cloud-based data and grid facilities. Various categories of cloud services have been utilized more and more widely across a variety of fields like military, army systems, medical databases, and more, in order to manage data storage and resource calculations. Attribute-based encipherment (ABE) is one of the more efficient algorithms that leads to better consignment and safety of information located within such cloud-based storage amenities. Many outmoded ABE practices are useful for smaller datasets to produce fixed-size cryptograms with restricted computational properties, in which their characteristics are measured as evidence and stagnant standards used to generate the key, encipherment, and decipherment means alike. To surmount the existing problems with such limited methods, in this work, a dynamic nonlinear poly randomized quantum hash system is applied to enhance the safety of cloud-based information. In the proposed work, users’ attributes are guaranteed with the help of a dynamic nonlinear poly randomized equation to initialize the chaotic key, encipherment, and decipherment. In this standard, structured and unstructured big data from clinical datasets are utilized as inputs. Real-time simulated outcomes demonstrate that the stated standard has superior exactness, achieving over 90% accuracy with respect to bit change and over 95% accuracy with respect to dynamic key generation, encipherment time, and decipherment time compared to existing models from the field and literature. Experimental results are demonstrated that the proposed cloud security standard has a good efficiency in terms of key generation, encoding, and decoding process than the conventional methods in a cloud computing environment.

Design of Unstructured Big Data Matching System for Unidentified Aircraft Tracking Using Open CV

In recent years, the spread of computers is increasing, and efficient processing effort for unstructured Big Data is required. In this paper, we are proposed a system to extract the data typed in a word processor quickly by user creating and XML mapping file after converting XML data that has been entered in the office file(HWP, MS-office). In addition, we proposed a system is able to lookup the necessary data from a database by entered form in advance and convert word processor document to office files by the application program. The unstructured big data will be available to be used.

Todays many devices generate data everywhere and anytime. Data grow massively and becomes complex thing that needs to be handled. Unstructured data is one type of big data that is difficult to process and consists of unstable attributes. In traffic management, CCTVs are installed to monitor the specific location in the highway. CCTV generates unstructured data in image and video format. These data are difficult to process due to the complexity of the data. This research proposes to implement big data analytics to process real-time unstructured data from CCTV into knowledge displayed in web dashboard. We implement the YOLO framework with YoloV4 Architecture and COCO dataset for traffic flow counting and detecting illegal parking which is categorized as abnormal situation. Unstructured data from CCTV then transformed into semi-structured format in JSON. Data also can be visualized in real time to facilitate local authority to understand the highway situation. Historical data are stored in the NoSQL database to deep more knowledge such as vehicle traffic pattern. The proposed system requires the ROI drawing line as trigger to count the passing vehicles. These experiments are conducted from open CCTV for traffic online in Bali Tower Public Streaming. The prototype result is able to detect the object with 10 fps.

There is an increasing need for both governments and businesses to discover latent anomalous activities in unstructured publicly-available data, produced by professional agencies and the general public. Over the past two decades, consumers have begun to use smart devices to both take in and generate a large volume of open-source text-based data, providing the opportunity for latent anomaly analysis. However, real-time data acquisition, and the processing and interpretation of various types of unstructured data, remains a great challenge. Recent efforts have focused on artificial intelligence / machine learning (AI/ML) solutions to accelerate the labor-intensive linear collection, exploitation, and dissemination analysis cycle and enhance it with a data-driven rapid integration and correlation process of open-source data. This paper describes an Activity Based Intelligence framework for anomaly detection of open-source big data using AI/ML to perform semantic analysis. The proposed Anomaly Detection using Semantic Analysis Knowledge (ADUSAK) framework includes four layers: input layer, knowledge layer, reasoning layer, and graphical user interface (GUI)/output layer. The corresponding main technologies include: Information Extraction, Knowledge Graph (KG) construction, Semantic Reasoning, and Pattern Discovery. Finally, ADUSAK was verified by performing Emerging Events Detection, Fake News Detection, and Suspicious Network Analysis. The generalized ADUSAK framework can be easily extended to a wide range of applications by adjusting the data collection, modeling construction, and event alerting.