Mass data processing and personalized services in Shanghai e-commerce credit evaluation platform

In the past three decades, with the fast development of China's economy, the business data of electric power industry also showed rapid growth and the data size is continual expanding. Thus, the management systems of electric enterprise groups encountered the performance bottlenecks of massive data processing. Hadoop is an open-source software framework developed for reliable, scalable, and efficient distributed computing and storage, and is widely used in massive data processing. Based on the study of the proposed massive business data processing of electric industry, this paper introduces a novel architecture, which is based on Hadoop ecosystem, for management systems of electric enterprise groups to crack the present performance bottlenecks. First, the business logic and processing produce of management system is analyzed. Then new architecture with Hadoop based distributed processing model is introduced in detailed. At last, experimental results with actual business scenarios and data prove the architecture can effectively resolve present performance bottlenecks of massive business data processing, and greatly improve the performance and efficiency of the management system of electric enterprise groups.

Chapter 8 - Managing the Cloud

This article delves into the communication quality monitoring system driven by big data under the C/S (client/server) architecture, aiming to provide a comprehensive, real-time, and efficient security monitoring solution for factory production and manufacturing environments. The system integrates LoRa wireless communication technology, big data processing and analysis technology, and innovative dynamic expansion algorithms to achieve real-time monitoring and rapid response to various safety hazards in the production environment. The core of the system lies in the construction of LoRa wireless communication network, which adopts an innovative umbrella shaped network structure to ensure stable and fast transmission of sensor monitoring data to the server. By using the STM32 development board with LoRa module, dynamic expansion of hardware integration modules has been achieved, enhancing the flexibility and scalability of the system. At the level of data transmission and processing, introducing Kafka distributed message queue as a data cache effectively alleviates the pressure of processing massive real-time data. Combining the Spark Streaming streaming data processing framework, a distributed data processing model was constructed to achieve real-time consumption and efficient parallel processing of messages in Kafka queues. At the same time, the designed dynamic extension algorithm model can automatically persist the data of new monitoring points, ensuring the system's rapid adaptation to data changes. In terms of data storage and visualization, a data persistence layer architecture for a big data platform has been constructed. Real time data, historical data, and log data are stored separately in Redis, MySQL, and HDFS systems through a data streaming model, improving data storage efficiency and system performance. Based on Tomcat network server and SSM architecture, a B/S structured web server visualization platform has been developed, which supports users to remotely query the security status of the production environment through computer or mobile browser, and receive alarm notifications in case of abnormalities. The system integration test results show that the communication quality monitoring system performs excellently in wireless communication quality, data processing speed, data storage efficiency, and data visualization, and can meet the complex monitoring needs of factory production and manufacturing environments, providing solid technical support for enterprise safety production. However, there is still room for improvement in real-time data collection loss rate, alarm function completeness, and mobile access experience. Future research will focus on optimizing data transmission software models, adding multi-channel alarm functions, and developing mobile apps to enhance user experience.

High-resolution topography data acquired with LIDAR (Light Detection and Ranging) remote sensing technology have emerged as a fundamental tool for Earth science research. Because these acquisitions are often undertaken with federal and state funds at significant cost, it is important to maximize the impact of these geospatial data by providing online access to a range of potential users. The National Science Foundation-funded OpenTopography Facility hosted at the San Diego Supercomputer Center (SDSC), has developed a Geospatial Cyberinfrastructure (GCI) to enable online access to Earth science-oriented high-resolution LIDAR topography data, online processing tools, and derivative products. Leveraging high performance computational and data storage resources available at SDSC, OpenTopography provides access to terabytes of point cloud data, standard digital elevation models, and Google Earth image data, all co-located with computational resources for higher-level data processing. This paper describes the motivation, goals, and the technical details of the Services Oriented Architecture (SOA) and underlying cyberinfrastructure platform implemented by OpenTopography. The use of an SOA, and the co-location of processing and data resources are unique to the field of LIDAR topography data processing, and lays a foundation for providing an open system for hosting and providing access to data and computational tools for these important scientific data, and is an exemplar for similar large geospatial data and processing community-oriented cyberinfrastructure systems.

Cloud security is one of the issues that cloud platform needs to focus on. Cloud platform has powerful computing ability and storage resources, which makes it become the target of hackers. Therefore, in addition to the traditional network security configuration, it is necessary to adopt a more complete security defense measure to protect the data processing platform. In this paper, it proposes a distributed anomaly traffic detection technology based on classifier combination according to the characteristics of massive network data processing in Hadoop platform, which can improve the security of massive network data processing platform.

SummaryBrain data processing has been embracing the big data era driven by the rapid advances of neuroscience as well as the experimental techniques for recording neuronal activities. Processing of massive brain data has become a constant in neuroscience research and practice, which is vital in revealing the hidden information to better understand the brain functions and malfunctions. Brain data are routinely non‐linear and non‐stationary in nature, and existing algorithms and approaches to neural data processing are generally complicated in order to characterize the non‐linearity and non‐stationarity. Brain big data processing has pressing needs for appropriate computing technologies to address three grand challenges: (1) efficiency, (2) scalability and (3) reliability. Recent advances of computing technologies are making non‐linear methods viable in sophisticated applications of massive brain data processing. General‐purpose Computing on the Graphics Processing Unit (GPGPU) technology fosters an ideal environment for this purpose, which benefits from the tremendous computing power of modern graphics processing units in massively parallel architecture that is frequently an order of magnitude larger than the modern multi‐core CPUs. This article first recaps significant speed‐ups of existing algorithms aided by GPGPU in neuroimaging and processing electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG) and etc. The article then demonstrates a series of successful approaches to processing EEG data in various dimensions and scales in a massively parallel manner: (1) decomposition: a massively parallel Ensemble Local Mean Decomposition (ELMD) algorithm aided by GPGPU can decompose EEG series, which forms the basis of further time‐frequency transformation, in real‐time without sacrificing the precision of processing; (2) synchronization measurement: a parallelized Nonlinear Interdependence (NLI) method for global synchronization measurement of multivariate EEG with speed‐up of more than 1000 times, and it was successful in localization of epileptic focus; and (3) dimensionality reduction: a large‐scale Parallel Factor Analysis which excels in run‐time performance and scales far better by hundreds of times than conventional approach does, and it supports fast factorization of EEG with more than 1000 channels. Through these practices, the massively parallel computing technology manifests great potentials in addressing the grand challenges of brain big data processing. Copyright © 2016 John Wiley & Sons, Ltd.

The FastICA algorithm and the natural gradient algorithm is widely used in blind signal separation, as the two more popular algorithms in the areas of the ICA, because they can find implicit in the observation data in the independent component. However, they each have drawback, such as, online data processing and mass data(such as image data)processing tasks, we need that FastICA algorithm is faster and more efficient; the same time we also hope that the nature gradient algorithm can not only deal with symmetric distribution of the signal, but also asymmetrical distribution of the signal. Therefore, this paper establishes a fifth-order convergence of the Newton iterative algorithm as a basis for improvement of the FastICA algorithm; farther more the paper also gives an improved natural gradient method which is used for symmetrical distribution of the signal and asymmetrical distribution of the signal.

Infrastructure as a Service is the pillar on which a cloud computing architecture is built. With the advancement of technologies in communications, computing, and storage devices, IaaS has emerged as a highly efficient platform to construct SaaS and PaaS layer on top of it. IaaS solutions vary from an organization to another. One single solution does not fit all. This chapter looks at the general constituents of IaaS.

Information management and related applications are being vigorously researched and developed. According to IDC research reports, the global market for information management is forecasted to grow by 30 percent annually (from a value of 8 billion US dollars in 2012, to a forecasted 24 billion US dollar value in 2016). Both reports explicitly identify the significant growth in the importance of information management and related services in the future. This paper will discuss and analyze our development in topic-specific applications, namely the collection, access, and enrichment of massive data. Through the development of adaptive file access technology and the combination of distributed storage (i.e. Hive, HBase, etc.) with In-Memory algorithmic technologies, we will be able to have the system adjust data storage policies and increase efficiency through the identification of data types and some algorithmic properties. Based on the virtual machine migration technology, we utilize the mass data processing workload to dynamically migrate the virtual computing node. This can automatically adjust virtual resources to achieve outstanding calculation and localization of data through parallel execution of multiple mass data management processes, and maximize massive data processing efficiency. In the future we plan to progressively seek potential clients and partners, from which we will tailor a variety of field tests according to our clients' needs. And through this process, is where we will gradually achieve technical and application verifications.

Number of internet connected devices is increasing as IoT is getting more prevalent. Volume of data collected by IoT sensors is very high and requires considerable resources for data processing like analytics. Edge processing enables getting the sensor data processed closer to the source. It's not always possible and/or economical to set up a resource intense infrastructure at the edge. This paper proposes use of blockchain based decentralized application to enable IoT edge processing. With this setup, it is possible for a resource owner to join the ecosystem and to lend the compute resources as needed. IoT devices can offload some of the edge computation to the resource owner nodes as and when required. This paper explains architectural components and overall solution for implementation. Based on the usecase of video analytics at edge, experimental setup is implemented for the IoT edge solution using hyperledger sawtooth blockchain.

In order to improve the control level of massive data transmission integrity, improve the space utilisation efficiency and data transmission integrity under cloud storage, an integrity control model of mass data transmission under big data cloud storage is constructed. Firstly, the basic principle of big data cloud storage is studied. The developing process of big data, data storage technology, and data retrieval technology is studied. Secondly, the mass data feature extraction model is established. The theory model and framework of feature extraction model for mass data are analysed. Thirdly, the massive data integrity control algorithm is designed based on an ant algorithm. The prior control models of single node rate are studied and system level parallel data transmission rate are analysed. Finally, simulation analysis is carried out, and the effectiveness of the proposed method is verified.

Understanding how proteins and materials interact is useful for evaluating the safety of biomedical micro/nanomaterials, toxicity estimation and design of nano-drugs and catalytic activity improvement of bio-inorganic functional hybrids. However, characterizing the interfacial molecular details of protein-micro/nanomaterial hybrids remains a great challenge. This protocol describes the lysine reactivity profiling-mass spectrometry strategy for determining which parts of a protein are interacting with the micro/nanomaterials. Lysine residues occur frequently on hydrophilic protein surfaces, and their reactivity is dependent on the accessibility of their amine groups. The accessibility of a lysine residue is lower when it is in contact with another object; allosteric effects resulting from this interaction might reduce or increase the reactivity of remote lysine residues. Lysine reactivity is therefore a useful indicator of protein localization orientation, interaction sequence regions, binding sites and modulated protein structures in the protein-material hybrids. We describe the optimized two-step isotope dimethyl labeling strategy for protein-material hybrids under their native and denaturing conditions in sequence. The comparative quantification results of lysine reactivity are only dependent on the native microenvironments of lysine local structures. We also highlight other critical steps including protein digestion, elution from materials, data processing and interfacial structure analysis. The two-step isotope labeling steps need ~5 h, and the whole protocol including digestion, liquid chromatography-tandem mass spectrometry, data processing and structure analysis needs ~3-5 d.

In aerospace launching mission, the requirements of the tracking and control system are frequently changing, which cause the high costs of software maintenance and low software reliability. To solve this problem, we research some key technologies in designing a reconfigurable real-time data process system. According to the characteristics of the tracking and control information, an information description model, a processing method model and a processing tree model are presented. Based on these models, re-configurable real-time data processing system architecture and resource configuration methods are proposed. The proposed methods improve the data processing system’s rapid adaption ability to new mission and new processing requirements.

With the continuous development of medical image informatics technology, more and more high-throughput quantitative data could be extracted from digital medical images, which has resulted in a new kind of omics-Radiomics. In recent years, in addition to genomics, proteomics and metabolomics, radiomic has attracted the interest of more and more researchers. Compared to other omics, radiomics can be perfectly integrated with clinical data, even with the pathology and molecular biomarker, so that the study can be closer to the clinical reality and more revealing of the tumor development. Mass data will also be generated in this process. Machine learning, due to its own characteristics, has a unique advantage in processing massive radiomic data. By analyzing mass amounts of data with strong clinical relevance, people can construct models that more accurately reflect tumor development and progression, thereby providing the possibility of personalized and sequential treatment of patients. As one of the cancer types whose treatment and diagnosis rely on imaging examination, radiomics has a very broad application prospect in head and neck cancers (HNC). Until now, there have been some notable results in HNC. In this review, we will introduce the concepts and workflow of radiomics and machine learning and their current applications in head and neck cancers, as well as the directions and applications of artificial intelligence in the treatment and diagnosis of HNC.

Optimal qubit mapping search for encoding classical data into matrix product state representation with minimal loss