Exploiting Sharing Join Opportunities in Big Data Multiquery Optimization with Flink

Modelling and analysis of big data platform group adoption behaviour based on social network analysis

The word Big Data is commonly used and it is not new today. Large, medium and small companies are starting to use Big Data to obtain their customers insight in order to serve them in a better way. The use of Big Data has become quite a crucial way for businesses to compete with their competitors. Also not only companies gain from the value of Big Data, it is also the customer’s hugely benefit from its usage. In association with Big Data’s real time information, which is one of the most heavily used application of personal and location data. As there is a significant growth in the use of smart phones and the use of GPS services from the phones and other devices, the use of smart traffic routing will definitely grow and in turn it will hugely benefit the customers. Big Data is not a single packaged technology, it is in general a platform consists of usage of various components to achieve a common goal. There are plenty of components available in the market for the businesses to customise their Big Data platform. The utilization of Big Data is becoming more and more essential to businesses and it is even more important for them to adopt the right Big Data platform to accomplish their goals. The main aim of this study is to propose a framework for building a Big Data platform for publishing industry. The proposed framework was validated in an UK based news publishing organisation to find out the suitability and adoptability of the framework for their Big Data platform.

Photo by Josh Riemer on Unsplash
 Introduction
 With the rapid advancements in neurotechnological machinery and improved analytical insights from machine learning in neuroscience, the availability of big brain data has increased tremendously. Neurological health research is done using digitized brain data.[1] There must be adequate data governance to secure the privacy of subjects participating in brain research and treatments. If not properly regulated, the research methods could lead to significant breaches of the subject’s autonomy and privacy. This paper will address the necessity for neuroprotection laws, which effectively govern the use of big brain data to ensure respect for patient privacy and autonomy.
 Background
 Artificial intelligence and machine learning can be integrated with neuroscience big brain data to drive research studies. This integrative technology allows patterns of electrical activity in neurons to be studied in detail.[2]Specifically, it uses a robotic system which can reason, plan, and exhibit biologically intelligent behavior. Machine learning is a method of computer programming where the code can adapt its behavior based on big brain data.[3] The big brain data is the collection of large amounts of information for the purpose of deciphering patterns through computer analysis using machine learning.[4] The information that these technologies provide is extensive enough to allow a researcher to read a patient’s mind. AI and machine learning technologies work by finding the underlying structure of brain data, which is then described by patterns known as latent factors, eventually resulting in an understanding of the brain’s temporal dynamics.[5]
 Through these technologies, researchers are able to decipher how the human brain computes its performances and thoughts. However, due to the extensive and complex nature of the data processed through AI and machine learning, researchers may gain access to personal information a patient may not wish to reveal. From a bioethical lens, tensions arise in the realm of patient autonomy. Patients are not able to control the transmission of data from their brains that is analyzed by researchers. Governing brain data through laws may enhance the extent of patient privacy in the case where brain data is being used through AI technologies.[6] A responsible approach to governing brain data would require a sophisticated legal structure.
 Analysis
 Impact on Patient Autonomy and Privacy 
 In research pertaining to big brain data, the consent forms do not fully cover the vast amounts of information that is collected. According to research, personal data has become the most sought out commodity to provide content to corporations and the web-based service industry. Unfortunately, data leaks that release private information frequently occur.[7] The storage of an individual’s data on technologies accessible on the internet during research studies makes it vulnerable to leaks, jeopardizing an individual’s privacy. These data leaks may cause the patient to be identified easily, as the degree of information provided by AI technologies are personalized and may be decoded through brain fingerprinting methods.[8]
 There has been an extensive growth in the development and use of AI. It is efficient in providing information to radiologists who diagnose various diseases including brain cancer and psychiatric disease, and AI assists in the delivery of telemedicine.[9] However, the ethical pitfall of reduced patient autonomy must be addressed by analyzing current AI technologies and creating more options for patient preference in how the data may be used. For instance, facial recognition technology[10] commonly used in health care produces more information than listed in common consent forms, threatening to undermine informed consent. Facial recognition software collects extensive data and may disclose more information than a person would prefer to provide despite being a useful tool for diagnosing medical and genetic conditions.[11] In addition, people may not be aware that their images are being used to generate more clinical data for other purposes. It is difficult to guarantee the data is anonymized. Consent requirements must include informing people about the complexity of the potential uses of the data; software developers should maximize patient privacy.[12] Furthermore, there is a “human element” in the use of AI technologies as medical providers control the use and the extent to which data is captured or accessed through the AI technologies.[13] People must understand the scope of the technology and have clear communication with the physician or health care provider about how the medical information will be used. 
 Existing Laws for Brain Data Governance 
 A strict system of defined legal responsibilities of medical providers will ensure a higher degree of patient privacy and autonomy when AI technologies and data from machine learning are used. Governing specific algorithmic data is crucial in safeguarding a patient’s privacy and developing a gold standard treatment protocol following the procurement of the information.[14] Certain AI technologies provide more data than others, and legal boundaries should be established to ensure strong performance, quality control, and scope for patient privacy and autonomy. For instance, currently AI technologies are being used in the realm of intensive neurological care. However, there is a significant level of patient uncertainty about how much control patients have over the data’s uses.[15] Calibrated legal and ethical standards will allow important brain data to be securely governed and monitored.
 Once brain signals are recorded and processed from one individual, the data may be merged with other data in Brain Computer Interface Technology (BCI).[16] To ensure a right and ability to retrieve personal data or pull it from the collection, specific regulations for varying types of data are needed.[17] The importance of consent and patient privacy must be considered through giving patients a transparent view of how brain data is governed.[18] The legal system must address discriminatory issues and risks to patients whose data is used in studies. Laws like the General Data Protection Regulation (GDPR) and the California Consumer Privacy Protection Act (CCPA) can serve as effective models to protect aggregated data. These laws govern consumer information and ensure the compliance when personal data is collected.[19] California voters recently approved expansion of the CCPA to health data. The Washington Privacy Act, which would have provided rights to access, change, and withdraw personal data, failed to pass. Other states should improve privacy as well,[20] although a federal bill would be preferable. Scientists at the Heidelberg Academy of Sciences argue for data security to be governed in a manner that balances patient privacy and autonomy with the commercial interests of researchers.[21] The balance could be achieved through privacy protections like those in the Washington Privacy Act. Although the Health Insurance Portability and Accountability Act (HIPAA) provides an overall framework to deter the likelihood of dangers to patient protection and privacy, more thorough laws are warranted to combat pervasive data transfer and analysis that technology has brought to the health care industry.[22] Breaches of patient privacy under current HIPAA regulations include releasing patient information to a reporter without their consent and sending HIV data to a patient’s employer without consent.[23] HIPAA does not cover information being shared with outside contractors who do not have an agreement with technology companies to keep patient data confidential. HIPAA regulations also do not always address blatant breaches on patient data confidentiality.[24] Patients must be provided with methods to monitor the data being analyzed to be able to view the extent of private information being generated via AI technologies. In health research, the medical purposes of better diagnosis, earlier detection of diseases, or prevention are ethical justifications for the use of the data if it was collected with permission, the person understood and approved the uses of the data, and the data was deidentified.
 A standard governance framework is required in providing the fairest system of care to patients who allow their brain data to be examined. Informed consent in the neuroscience field could reaffirm the privacy and autonomy of patients by ensuring that they understand the type of information collected. Laws also could protect data after a patient’s death. Malpractice in the scope of brain data could give people a cause of action critical in safeguarding patient’s rights. Data breach lawsuits will become common but generally do not cover deidentified data that becomes part of big data collection. A more synchronized approach to the collection and consent process will encourage an understanding of how big data is used to diagnose and treat patients. Some altruistic people may even be more likely to consent if they know the largescale data collection is helpful to treat and diagnose people. Others should have the ability to opt out of sharing neurological data, especially when there is not certainty surrounding deidentification.[25]
 Conclusion
 Artificial intelligence and machine learning technologies have the potential to aid in the diagnosis and treatment of people globally by extracting and aggregating brain data specific to individuals. However, the secure use of the data is necessary to build trust between care providers and patients, as well as in balancing the bioethical principles of beneficence and patient autonomy. We must ensure the highest quality of care to patients, while protecting their privacy, informed consent, and clinical trust. More sophis

In order to improve the efficiency of locomotive organization and the quality of locomotive operation, this paper analyzes and discusses the big data platform and some key technologies suitable for the big data application of the railway locomotive system. Firstly, the definition of big data of the railway locomotive system is proposed, and the current data characteristics of the railway locomotive system are summarized, then the status quo and demands of big data application of the railway locomotive system are analyzed. Secondly, the overall architecture of the big data platform for the railway locomotive system is proposed. Furthermore, seven application scenarios available for the big data platform are analyzed, including locomotive running organization, high-speed railway, repair, maintenance and other fields. Finally, some key technologies, which consist of data collection system of front-line operations, locomotive equipment portrait analysis, staff portrait analysis, transmission and analysis of locomotive video, intelligent auxiliary driving system, are provided to increase efficiency of the locomotive organization and capability of safety management. The obtained results can play a positive role in the construction and application of big data of the railway locomotive system.

China Unicom, the largest WCDMA 3G operator in China, meets the requirements of the historical Mobile Internet Explosion, or the surging of Mobile Internet Traffic from mobile terminals. According to the internal statistics of China Unicom, mobile user traffic has increased rapidly with a Compound Annual Growth Rate (CAGR) of 135%. Currently China Unicom monthly stores more than 2 trillion records, data volume is over 525 TB, and the highest data volume has reached a peak of 5 PB. Since October 2009, China Unicom has been developing a home-brewed big data storage and analysis platform based on the open source Hadoop Distributed File System (HDFS) as it has a long-term strategy to make full use of this Big Data. All Mobile Internet Traffic is well served using this big data platform. Currently, the writing speed has reached 1 390 000 records per second, and the record retrieval time in the table that contains trillions of records is less than 100 ms. To take advantage of this opportunity to be a Big Data Operator, China Unicom has developed new functions and has multiple innovations to solve space and time constraint challenges presented in data processing. In this paper, we will introduce our big data platform in detail. Based on this big data platform, China Unicom is building an industry ecosystem based on Mobile Internet Big Data, and considers that a telecom operator centric ecosystem can be formed that is critical to reach prosperity in the modern communications business.

The emergence of big data platforms has alleviated the restrictions of space and time on people's enjoyment of music information services. The need to improve user experience has become very urgent. One of the effective solutions to information overload is the recommendation system, which can help people discover interesting content from complex information. Therefore, the purpose of this article is to study a personalized film and television recommendation system based on a big data platform. This article first introduces the core technology of big data through an overview of the basic theory of big data. And combined with the traditional film and television recommendation method, analyze the existing problems and shortcomings, and analyze and discuss it with the core technology of big data. This article systematically expounds the overall process, system composition design and algorithm analysis of the personalized film and television recommendation system based on the big data platform. Experimental research shows that the algorithm proposed in this paper is better than the traditional User-basedCF strategy and Item-basedCF strategy under the same number of movies recommended by Hit-rate.

It is of great significance to establish a scientific and perfect application model of medical and health big data analysis for promoting the application of medical and health big data. This paper mainly studies the computer-aided medical and health data analysis of Inner Mongolia Autonomous Region based on big data. This paper adopts literature research method, field research method and data science method. Literature research is used to understand big data platforms and their technologies, and to analyze the limitations of data analysis methods in traditional data warehouses. The application process of data analysis based on big data platform is studied by using data science method. The established medical and health data analysis application process based on the big data platform and the described medical and health big data application theme were consulted by using the expert consultation method, and they were modified and improved according to the expert opinions.KeywordsBig data platformComputer aidedHealthcare dataData analysis

The rapid growth of big data has created new opportunities for developing intelligent systems capable of supporting effective decision-making processes. In this context, recommendation systems play a crucial role by providing personalized and data-driven suggestions to users. However, traditional recommendation approaches often face limitations in scalability, accuracy, and the ability to process large-scale heterogeneous data. This paper focuses on the design of an intelligent recommendation system based on big data analytics. The study analyzes modern big data technologies and machine learning methods used for collecting, processing, and analyzing large volumes of structured and unstructured data. A system architecture is proposed that integrates data preprocessing techniques, big data platforms, and intelligent algorithms to improve recommendation quality. Experimental results demonstrate that the proposed approach enhances recommendation accuracy and system performance compared to conventional methods. The findings highlight the effectiveness of big data–driven analytics in developing scalable and intelligent recommendation systems. The results of this study can be applied in various domains such as e-commerce, education, and decision support systems, and provide a foundation for future research in intelligent data-driven system design.

The rapid growth of new computing paradigms such as Cloud Computing and Big Data has unleashed great opportunities for companies to shift their business model towards a fully digital strategy. A major obstacle in this matter is the requirement of highly specialized ICT infrastructures that are expensive and difficult to manage. It is at this point that the IaaS (infrastructure as a service) model offers an efficient and cost‐affordable solution to supply companies with their required computing resources. In the Big Data context, it is often a hard task to design an optimal IaaS solution that meets user requirements. In this context, we propose a methodology to optimize the definition of IaaS cloud models for hosting Big Data platforms, following a threefold criterion: cost, reliability, and computing capacity. Specifically, the proposed methodology hinges on evolutionary heuristics in order to find IaaS configurations in the cloud that optimally balance such objectives. We also define measures to quantify the aforementioned metrics over a Big Data platform hosted within an IaaS cloud model. The proposed method is validated by using real information from three IaaS providers and three Big Data platforms. The obtained results provide an insightful input for system managers when initially designing cloud infrastructures for Big Data applications.

Big data is a term that is used to describe data that is high volume, high velocity, and/or high variety; requires new technologies and techniques to capture, store, and analyze it; and is used to enhance decision making, provide insight and discovery, and support and optimize processes. With regard to the definition of big data, IBM Company uses volume, velocity, variety, value and veracity as 5Vs to summarize the concept of big data. There are different types of big data, for example, structured, semi-structured and un-structured data. The contents of big data can be text data, audio data, video data and still image and it indicates that the big data may have diverse data types as well as data qualities. Big data has variety of sources such as healthcare center, commercial system, industries, social media, telecommunication, transportation, sensor machines and others. In this paper, I reviewed three the most security challenging perspectives and I studied lack of concentrations in these areas by most research works. To confirm security in the big data platforms, it is critical to ascertain the data rendering points and their security techniques to safeguard the data in this pacing digital world. Then I envisage directions for the future research. In this paper, I have reviewed the big data sources and its security issues in the three directions such as data at rest, data at communication and data in process/use. Keywords: Big Data, Big Data Security, Big Data source, Attribute based encryption, storage path, Transport layer security, secure shell, Cloud service Provider DOI: 10.7176/CEIS/12-3-01 Publication date: November 30 th 2021

The refinement and rationalization of urban design has become an important trend of restructuring and development during the crucial stage of China's urban transformation. At the same time, big data technology is developing rapidly, with the availability of various types of high-precision data being gradually increased. The combination of big data and urban design plays an important role in the urban design of transition period. This paper describes the design of urban education system under the big data platform. With the traditional teaching model, which include four stage (research, topic, design and expression), big data technologies are divided into four aspects to consider how to apply the big data to the teaching of urban design. Specifically summarized: big data accelerate researching, big data secondary analysis, big data enhanced design and big data of optimization expression. The aim of this research is to let the traditional teaching system and cutting-edge technology of large data collide with each other fusion, which can not only enrich the existing education system, but also establish a complete technical structure. On this basis, it can help students establish and improve the construction, and the ability to learn new technologies, to promote the progress of urban design techniques.

Based on information asymmetry theory, principal‐agent theory, and risk management theory, this paper analyzes the mechanism of the impact of big data on financial decision‐making, analyzing four dimensions: how big data enhances the information base for forecasting, how big data improves the relevance of decision‐making, how big data builds new competitive advantages, and how big data promotes dynamic decision‐making. Secondly, through the analysis of specific implementation cases of enterprise big data in financial decision‐making, we focus on the real problems faced in management and the effect of solving problems through big data platform. The enterprise realizing the integration of business and finance not only can better lead business expansion, but also can improve the management level within the enterprise, which is conducive to the improvement of core competitiveness. The integration of industry and finance is essentially achieved through the application of various financial management modules to the business operations of enterprises, including budget management, capital management, fixed asset management, and financial accounting. If we want to implement the whole process of financial integration, it is impossible to achieve this manually, and we must first build a powerful information system as a guarantee. Under the guidance of theories of information asymmetry, stakeholders, and management information systems, Internet finance enterprises should build their own integrated business finance system based on the demand for business finance integration in the Internet finance industry, to enhance the matching of business finance data of Internet finance enterprises, improve the efficiency of enterprise management, and realize business finance integration. Finally, through the research of this paper, we hope to provide reference for other similar enterprises to apply big data for financial decision‐making. Through the application of big data, higher economic benefits are achieved in procurement management, production control, capital budget, and investment decision compared with the previous ones. It is concluded that in the era of big data, massive data can be used to serve enterprise decision‐making in depth, which can break the business and financial barriers, improve the efficiency and quality of decision‐making, optimize the organizational structure and personnel, and enhance the prediction and early warning capability. The application of big data tools has become the key to assisting financial decision‐making and enhancing enterprise value.

Data stream management systems usually have to process many long-running queries that are active at the same time. Multiple queries can be evaluated more efficiently together than independently, because it is often possible to share state and computation. Motivated by this observation, various Multi-Query Optimization (MQO) techniques have been proposed. However, these approaches suffer from two limitations. First, they focus on very specialized workloads. Second, integrating MQO techniques for CQL-style stream engines and those for event pattern detection engines is even harder, as the processing models of these two types of stream engines are radically different.

Recently the topic of how to effectively offload cellular traffic onto device-to-device (D2D) sharing among users in proximity has been gaining more and more attention of global researchers and engineers. Users utilize wireless short-range D2D communications for sharing contents locally, due to not only the rapid sharing experience and free cost, but also high accuracy on deliveries of interesting and popular contents, as well as strong social impacts among friends. Nevertheless, the existing related studies are mostly confined to small-scale datasets, limited dimensions of user features, or unrealistic assumptions and hypotheses on user behaviors. In this article, driven by emerging Big Data techniques, we propose to design a big data platform, named D2D Big Data, in order to encourage the wireless D2D communications among users effectively, to promote contents for providers accurately, and to carry out offloading intelligence for operators efficiently. We deploy a big data platform and further utilize a large-scale dataset (3.56 TBytes) from a popular D2D sharing application (APP), which contains 866 million D2D sharing activities on 4.5 million files disseminated via nearly 850 million users in 13 weeks. By abstracting and analyzing multidimensional features, including online behaviors, content properties, location relations, structural characteristics, meeting dynamics, social arborescence, privacy preservation policies and so on, we verify and evaluate the D2D Big Data platform regarding predictive content propagating coverage. Finally, we discuss challenges and opportunities regarding D2D Big Data and propose to unveil a promising upcoming future of wireless D2D communications.

The approach to improvement of performance of distributed information systems based on sharing technologies of the Hadoop cluster and component of SQL Server PolyBase was considered. It was shown that the relevance of the problem, solved in the research, relates to the need for processing Big Data with different way of representation, in accordance with solving diverse problems of business projects. An analysis of methods and technologies of creation of hybrid data warehouses based on different data of SQL and NoSQL types was performed. It was shown that at present, the most common is the technology of Big Data processing with the use of Hadoop distributed computation environment. The existing technologies of organization and access to the data in the Hadoop cluster with SQL-like DBMS by using connectors were analyzed. The comparative quantitative estimates of using Hive and Sqoop connectors during exporting data to the Hadoop warehouse were presented. An analysis of special features of Big Data processing in the architecture of Hadoop-based distributed cluster computations was carried out. The features of Polybase technology as a component of SQL Server for organizing a bridge between SQL Server and Hadoop data of the SQL and NoSQL types were presented and described. The composition of the model computer plant based on the virtual machine for implementation of joint setting of PolyBase and Hadoop for solving test tasks was described. A methodological toolset for the installation and configuration of Hadoop and PolyBase SQL Server software was developed with consideration of constraints on computing capacities. Queries for using PolyBase and data warehouse Hadoop when processing Big Data were considered. To assess the performance of the system, absolute and relative metrics were proposed. For large volume of test data, the results of the experiments were presented and analyzed, which illustrated an increase in productivity of the distributed information system – query execution time and magnitude of memory capacity of temporary tables, created in this case. A comparative analysis of the studied technology with existing connectors with Hadoop cluster, which showed the advantage of PolyBase over connectors of Sqoop and Hive was performed. The results of the research could be used in the course of scientific and training experiments of organization when implementing the most modern IT-technologies.