Design of big data integration platform based on hybrid hierarchy architecture

Objective: Through the practice discussion and exchange of the integration of big data and pharmaceutical standards, to provide experience reference for the practice of information technology in the management of rational drug use. Methods: The application of data information in the practice of hospital pharmacy management is demonstrated through the introduction of practical cases and data achievements at different levels. Results: The integration of medical big data and pharmaceutical standards effectively improved the hospital's quantitative quality management indicators and the informatization of many pharmaceutical management methods in the hospital is realized. Conclusions: The integration of medical big data and pharmaceutical standards can effectively help improve the management level of rational drug use in hospitals, and there are still many possibilities for future exploration.

Big Data projects often require data from several sources to be integrated before they can be used for analysis. Once data have been integrated, they allow more detailed analysis that would otherwise not be possible. Accordingly, recent years have seen an increasing interest in techniques that facilitate the integration of data from diverse sources. Whenever data about individuals, or otherwise sensitive data, are to be integrated across organizations, privacy and confidentiality have to be considered. Domains where privacy preservation during data integration is of importance include business collaborations, health research, national censuses, the social sciences, crime and fraud detection, and homeland security. Increasingly, applications in these domains require data from diverse sources (both internal and external to an organization) to be integrated.Consequently, in the past decade, various techniques have been developed that aim to facilitate data integration without revealing any private or confidential information about the databases and records that are integrated. These techniques either provably prevent leakage of any private information, or they provide some empirical numerical measure of the risk of disclosure of private information.In the first part of this presentation we provide a background on data integration, and illustrate the importance of preserving privacy during data integration with several application scenarios. We then given an overview of the main concepts and techniques that have been developed to facilitate data integration in such ways that no private or confidential information is being revealed. We focus on privacy-preserving record linkage (PPRL), where so far most research has been conducted. We describe the basic protocols used in PPRL, and several key technologies employed in these protocols. Finally, we discuss the challenges privacy poses to data integration in the era of Big Data, and we discuss directions and opportunities in this research area.

Photo by Josh Riemer on Unsplash
 Introduction
 With the rapid advancements in neurotechnological machinery and improved analytical insights from machine learning in neuroscience, the availability of big brain data has increased tremendously. Neurological health research is done using digitized brain data.[1] There must be adequate data governance to secure the privacy of subjects participating in brain research and treatments. If not properly regulated, the research methods could lead to significant breaches of the subject’s autonomy and privacy. This paper will address the necessity for neuroprotection laws, which effectively govern the use of big brain data to ensure respect for patient privacy and autonomy.
 Background
 Artificial intelligence and machine learning can be integrated with neuroscience big brain data to drive research studies. This integrative technology allows patterns of electrical activity in neurons to be studied in detail.[2]Specifically, it uses a robotic system which can reason, plan, and exhibit biologically intelligent behavior. Machine learning is a method of computer programming where the code can adapt its behavior based on big brain data.[3] The big brain data is the collection of large amounts of information for the purpose of deciphering patterns through computer analysis using machine learning.[4] The information that these technologies provide is extensive enough to allow a researcher to read a patient’s mind. AI and machine learning technologies work by finding the underlying structure of brain data, which is then described by patterns known as latent factors, eventually resulting in an understanding of the brain’s temporal dynamics.[5]
 Through these technologies, researchers are able to decipher how the human brain computes its performances and thoughts. However, due to the extensive and complex nature of the data processed through AI and machine learning, researchers may gain access to personal information a patient may not wish to reveal. From a bioethical lens, tensions arise in the realm of patient autonomy. Patients are not able to control the transmission of data from their brains that is analyzed by researchers. Governing brain data through laws may enhance the extent of patient privacy in the case where brain data is being used through AI technologies.[6] A responsible approach to governing brain data would require a sophisticated legal structure.
 Analysis
 Impact on Patient Autonomy and Privacy 
 In research pertaining to big brain data, the consent forms do not fully cover the vast amounts of information that is collected. According to research, personal data has become the most sought out commodity to provide content to corporations and the web-based service industry. Unfortunately, data leaks that release private information frequently occur.[7] The storage of an individual’s data on technologies accessible on the internet during research studies makes it vulnerable to leaks, jeopardizing an individual’s privacy. These data leaks may cause the patient to be identified easily, as the degree of information provided by AI technologies are personalized and may be decoded through brain fingerprinting methods.[8]
 There has been an extensive growth in the development and use of AI. It is efficient in providing information to radiologists who diagnose various diseases including brain cancer and psychiatric disease, and AI assists in the delivery of telemedicine.[9] However, the ethical pitfall of reduced patient autonomy must be addressed by analyzing current AI technologies and creating more options for patient preference in how the data may be used. For instance, facial recognition technology[10] commonly used in health care produces more information than listed in common consent forms, threatening to undermine informed consent. Facial recognition software collects extensive data and may disclose more information than a person would prefer to provide despite being a useful tool for diagnosing medical and genetic conditions.[11] In addition, people may not be aware that their images are being used to generate more clinical data for other purposes. It is difficult to guarantee the data is anonymized. Consent requirements must include informing people about the complexity of the potential uses of the data; software developers should maximize patient privacy.[12] Furthermore, there is a “human element” in the use of AI technologies as medical providers control the use and the extent to which data is captured or accessed through the AI technologies.[13] People must understand the scope of the technology and have clear communication with the physician or health care provider about how the medical information will be used. 
 Existing Laws for Brain Data Governance 
 A strict system of defined legal responsibilities of medical providers will ensure a higher degree of patient privacy and autonomy when AI technologies and data from machine learning are used. Governing specific algorithmic data is crucial in safeguarding a patient’s privacy and developing a gold standard treatment protocol following the procurement of the information.[14] Certain AI technologies provide more data than others, and legal boundaries should be established to ensure strong performance, quality control, and scope for patient privacy and autonomy. For instance, currently AI technologies are being used in the realm of intensive neurological care. However, there is a significant level of patient uncertainty about how much control patients have over the data’s uses.[15] Calibrated legal and ethical standards will allow important brain data to be securely governed and monitored.
 Once brain signals are recorded and processed from one individual, the data may be merged with other data in Brain Computer Interface Technology (BCI).[16] To ensure a right and ability to retrieve personal data or pull it from the collection, specific regulations for varying types of data are needed.[17] The importance of consent and patient privacy must be considered through giving patients a transparent view of how brain data is governed.[18] The legal system must address discriminatory issues and risks to patients whose data is used in studies. Laws like the General Data Protection Regulation (GDPR) and the California Consumer Privacy Protection Act (CCPA) can serve as effective models to protect aggregated data. These laws govern consumer information and ensure the compliance when personal data is collected.[19] California voters recently approved expansion of the CCPA to health data. The Washington Privacy Act, which would have provided rights to access, change, and withdraw personal data, failed to pass. Other states should improve privacy as well,[20] although a federal bill would be preferable. Scientists at the Heidelberg Academy of Sciences argue for data security to be governed in a manner that balances patient privacy and autonomy with the commercial interests of researchers.[21] The balance could be achieved through privacy protections like those in the Washington Privacy Act. Although the Health Insurance Portability and Accountability Act (HIPAA) provides an overall framework to deter the likelihood of dangers to patient protection and privacy, more thorough laws are warranted to combat pervasive data transfer and analysis that technology has brought to the health care industry.[22] Breaches of patient privacy under current HIPAA regulations include releasing patient information to a reporter without their consent and sending HIV data to a patient’s employer without consent.[23] HIPAA does not cover information being shared with outside contractors who do not have an agreement with technology companies to keep patient data confidential. HIPAA regulations also do not always address blatant breaches on patient data confidentiality.[24] Patients must be provided with methods to monitor the data being analyzed to be able to view the extent of private information being generated via AI technologies. In health research, the medical purposes of better diagnosis, earlier detection of diseases, or prevention are ethical justifications for the use of the data if it was collected with permission, the person understood and approved the uses of the data, and the data was deidentified.
 A standard governance framework is required in providing the fairest system of care to patients who allow their brain data to be examined. Informed consent in the neuroscience field could reaffirm the privacy and autonomy of patients by ensuring that they understand the type of information collected. Laws also could protect data after a patient’s death. Malpractice in the scope of brain data could give people a cause of action critical in safeguarding patient’s rights. Data breach lawsuits will become common but generally do not cover deidentified data that becomes part of big data collection. A more synchronized approach to the collection and consent process will encourage an understanding of how big data is used to diagnose and treat patients. Some altruistic people may even be more likely to consent if they know the largescale data collection is helpful to treat and diagnose people. Others should have the ability to opt out of sharing neurological data, especially when there is not certainty surrounding deidentification.[25]
 Conclusion
 Artificial intelligence and machine learning technologies have the potential to aid in the diagnosis and treatment of people globally by extracting and aggregating brain data specific to individuals. However, the secure use of the data is necessary to build trust between care providers and patients, as well as in balancing the bioethical principles of beneficence and patient autonomy. We must ensure the highest quality of care to patients, while protecting their privacy, informed consent, and clinical trust. More sophis

Biological data are generated at unprecedentedly exponential rates, posing considerable challenges in big data deposition, integration and translation. The BIG Data Center, established at Beijing Institute of Genomics (BIG), Chinese Academy of Sciences, provides a suite of database resources, including (i) Genome Sequence Archive, a data repository specialized for archiving raw sequence reads, (ii) Gene Expression Nebulas, a data portal of gene expression profiles based entirely on RNA-Seq data, (iii) Genome Variation Map, a comprehensive collection of genome variations for featured species, (iv) Genome Warehouse, a centralized resource housing genome-scale data with particular focus on economically important animals and plants, (v) Methylation Bank, an integrated database of whole-genome single-base resolution methylomes and (vi) Science Wikis, a central access point for biological wikis developed for community annotations. The BIG Data Center is dedicated to constructing and maintaining biological databases through big data integration and value-added curation, conducting basic research to translate big data into big knowledge and providing freely open access to a variety of data resources in support of worldwide research activities in both academia and industry. All of these resources are publicly available and can be found at http://bigd.big.ac.cn.

This study attempts to build a big data platform for disease burden that can realize the deep coupling of artificial intelligence and public health. This is a highly open and shared intelligent platform, including big data collection, analysis, and result visualization. Based on data mining theory and technology, the current situation of multisource data on disease burden was analyzed. Putting forward the disease burden big data management model, functional modules, and technical framework, Kafka technology is used to optimize the transmission efficiency of the underlying data. This will be an efficient and highly scalable data analysis platform through embedding embedded Sparkmlib in the Hadoop ecosystem. With the concept of "Internet + medical integration," the overall architecture design of the big data platform for disease burden management was proposed based on the Spark engine and Python language. The main system composition and application scenarios are given at four levels: multisource data collection, data processing, data analysis, and the application layer, according to application scenarios and use requirements. The big data platform of disease burden management helps to promote the multisource convergence of disease burden data and provides a new path for the standardized paradigm of disease burden measurement. Provide methods and ideas for the deep integration of medical big data and the formation of a broader standard paradigm.

Aiming at the low cleaning rate of the traditional multi-source heterogeneous power grid big data cleaning model, a multi-source heterogeneous power grid big data cleaning model based on machine learning classification algorithm is designed. By capturing high-quality multi-source heterogeneous power grid big data, weight labeling of data source importance measurement, data attributes and tuples, and constructing Tan network based on the idea of machine learning classification algorithm, the data probability value is finally used to complete the classification and cleaning of inaccurate data. Experiments show that the model based on machine learning classification algorithm can effectively improve the imprecise data cleaning rate compared with the traditional model to solve multi-source heterogeneous imprecise data cleaning.

With the rise of heterogeneous large spatial and non-spatial data, systems are developed to manage these data sets. Big data emphasizes heterogeneity among systems leading to data integration issues due to the nature of big data which includes volume, variety and velocity. MyLake is a National Lake Database to manage information and knowledge sharing on lakes in Malaysia. At the moment, data are uploaded by each agency using MyLake as a platform. Nevertheless, this is carried out manually and require timely human effort. Each agency does one-to-one data integration (in-silo) where the integration is developed according to the agency-specific needs resulting from a possibility of integration issue across agencies. Therefore, this paper introduces a big data integration approach that extends the MyLake repository. The big data integration platform is a preliminary idea of how data can be shared, integrated, retrieved, and disseminated within a reliable and authenticated environment. The proposed centralized platform consisting of a set of standards, tools, repository and registry that enable multiple integrations between different agencies. The platform offers the potential to provide a reliable platform that acts as data retriever and disseminator.

With the rapid development of smart grid technology, a large amount of multi-source heterogeneous data has been generated in the power system, and its effective utilization is crucial for the optimization operation, demand prediction, and anomaly detection of the power system. However, the fusion processing of multi-source heterogeneous data faces many challenges, such as inconsistent data format, granularity, and quality, and direct fusion can easily lead to information redundancy and contradictions. A multi-source heterogeneous data fusion technology based on big data mining has been proposed to address the above issues. This method combines the advantages of convolutional neural networks and gated recurrent units to automatically extract features from image and sequence data and handle long-term dependency issues in time series data. Meanwhile, the K-means clustering algorithm is used to preprocess the data and train a specialized ConvGRU model. The results showed that in short-term load forecasting and abnormal electricity consumption behaviour detection tasks, the accuracy of this method reached 96.3% and 98.7%, respectively, with AUC values of 0.994 and 0.996. Compared to models that use only CNN or GRU, the performance is significantly improved. This method effectively solves the problem of integrating and processing multi-source heterogeneous power data, improves the accuracy and efficiency of power system data analysis, and provides strong support for the optimized operation of smart grids.

Taking a byte out of big data

An Analysis on Future Prospects of Big Data in Healthcare

Advances in the field of technology enabled individuals and businesses to collect large amounts of data (structured and unstructured) from various sources like never before. Data from social media, user-generated, internet, health care, manufacturing, supply chain, financial institution, and sensors have grown exponentially. This paper’s objective is to review how big data drive and impact innovation, competitive advantage, productivity, and decision support. Methodology: A comprehensive literature review on big data and identifying the impact of big data analytics on innovation, competitive advantage, productivity, and decision support are studied. The reviewed literature created the foundation for studying, a model that was developed based on an extensive review of literature as well as case studies and future forecast by market leaders. Big data is the latest buzzword among businesses. A new model is suggested identifying big data and the correlation between innovation, competitive advantage, productivity, and decision support. Findings: A review of scholarly literature and existing case studies finds that there is a gap between existing frameworks and the integration of big data into various business and management functions and objectives. The findings are interesting that literature is rich with concepts and frameworks for achieving the end goal for business or management function along with framework but very little is available in the literature on the question of how to integrate big data analytics into those frameworks. This paper finds that a key question is missing i.e., what are the essential steps that businesses should perform to implement and integrate big data analytics into existing frameworks to fully exploit the big data potential. Research Limitations/Implications: The research was limited to a review of selective literature focused on in-depth understanding of big data. Additionally, it focuses on how big data leads to innovation, competitive advantage, productivity, and decision support. Although there are many other related fields of studies where big data impact can be studied but is not part of this study effort. Future studies can lead to more in-depth studies of other related areas of studies. Practical Implications: The review of the literature suggested that “Big Data” is playing an important role in innovations, creating competitive advantage, enhancing productivity, and assisting in data-driven decisions. Businesses are taking advantage of the customer insights that are innovating products and services which are very customer-centric, keeping the competition on the run, improving productivity at all levels, and making educated decisions every day. The future will be driven by smarter big data solutions and insights. Originality/Value: The study provides evidence that big data is the catalyst for innovation, creates competitive advantage, enhances productivity, and assists in decision making. The methodology is to review scholarly literature and case studies. It supports the need for developing new models, implementation frameworks for better insights, and patterns. The big data implementation methodologies, framework, and governance have been ignored in empirical research.

IoT is upgrading and evolving due to its smart features and comfortless, whereas big data also helps storing and managing a huge amount of data effectively. In this paper we are trying to analyze the data created by IoT to provide an application and how effectively be managed by big data despite the data being structured, semi-structured or unstructured and while doing the management how cloud-based platform-as-a-service system (PaaS) is useful for a durable and cost-effective storage application. Later part of this paper we also discussed the various aspects and challenges of IoT may encounter while providing the services with respect to the volume, variety and velocity. We also suggested some techniques to resolve these problems for a better IoT-based system with integration of big data.

Stimulated by the arrival of the big data era, various and heterogeneous data sources such as data in social networks, mobile devices and sensor data for users have emerged, mirroring characteristics and preferences of data owners. These data sources are often used to construct user profiles so as to facilitate personalized services like recommendations or personalized data access. In the context of second language learning, learner data involve learning logs, standard test results, and individual learning preferences and styles. Given its attribute of reflecting the characteristics of learners, such data can be exploited to build the learner profiles. However, these data sources possibly include noises or bias, and hence influence the reliability of the correspondingly constructed learner profiles. Consequently, the inaccurate profiles may result in ineffective learning tasks that are generated by e-Learning systems. To tackle this issue, it is significant and critical to evaluate the accuracy of learner profiles. In a response to this call, we propose a novel metric named “profile mean square error” to examine the accuracy of learner profiles founded upon diverse sources. We also demonstrate how to construct various learner profiles though applying different data sources such as learning logs, standard test results, and personal learning preferences in e-Learning systems and pedagogical activities. Moreover, we conduct an experimental study among some second language learners, the results of which illustrate that the most accurate profiles are generated from multiple data sources if they are integrated in a rational way.

Industrial big data analysis and mining are extremely complicated since it has complex correlations and heterogeneous structure from multiple data source. The growing industrial big data makes data analysis and mining extremely complicated. However, the traditional analysis approach based on relational databases or data warehouses are not flexible enough to deal with multi-source heterogeneous data and are less efficient to do search and analysis operation. Based on Spark and Elasticsearch, this paper presents a multi-dimensional analysis method and system for industrial big data. An OLAP model architecture based on JSON document structure is proposed, which can use Key-Value structure to flexibly define diverse industrial data, and the multi-dimensional structure model is easy to query and analyze. The table structure in the dimension information is converted into a JSON-based document structure, and the dimension information contained in the fact table is stored by the nested document. Elasticsearch is used to store the document structure tree and build an inverted index, which can improve the efficiency of the data analysis query. The query and analysis operations are transformed into the traversal and query operations in the document content. The time efficiency of the multi-dimensional analysis system based on Elasticsearch is much better than the analysis efficiency based on Hive.

Imagine being able to target an audience made up of highly qualified and purchase-ready prospects and easily building them into loyal clients by anticipating their needs and hence offering true value. This is the power of big data for digital marketing. Big Data: A Roadmap for Successful Digital Marketing explores recent trends in the use of big data to predict consumer behavior, strategies to engage online customers, integration of big data with other data sources, and its applications in social media analytics, mobile marketing, search engine optimization and customer relationship management. As the marketing world moves into a data-focused future, the success of marketing efforts will be wholly based on attention to detail in data analysis and effectively acting on insights in order to implement changes that will deliver improved results. This book will help professionals succeed in their digital marketing efforts as well as provide food for thought for students and researchers in the fields of digital marketing, customer behavior and big data analytics.