Research of baseline solution batch processing algorithm for high precision GPS network with massive data

Overlay analysis is a common task in geographic computing that is widely used in geographic information systems, computer graphics, and computer science. With the breakthroughs in Earth observation technologies, particularly the emergence of high-resolution satellite remote-sensing technology, geographic data have demonstrated explosive growth. The overlay analysis of massive and complex geographic data has become a computationally intensive task. Distributed parallel processing in a cloud environment provides an efficient solution to this problem. The cloud computing paradigm represented by Spark has become the standard for massive data processing in the industry and academia due to its large-scale and low-latency characteristics. The cloud computing paradigm has attracted further attention for the purpose of solving the overlay analysis of massive data. These studies mainly focus on how to implement parallel overlay analysis in a cloud computing paradigm but pay less attention to the impact of spatial data graphics complexity on parallel computing efficiency, especially the data skew caused by the difference in the graphic complexity. Geographic polygons often have complex graphical structures, such as many vertices, composite structures including holes and islands. When the Spark paradigm is used to solve the overlay analysis of massive geographic polygons, its calculation efficiency is closely related to factors such as data organization and algorithm design. Considering the influence of the shape complexity of polygons on the performance of overlay analysis, we design and implement a parallel processing algorithm based on the Spark paradigm in this paper. Based on the analysis of the shape complexity of polygons, the overlay analysis speed is improved via reasonable data partition, distributed spatial index, a minimum boundary rectangular filter and other optimization processes, and the high speed and parallel efficiency are maintained.

New completion time algorithms for sequence based scheduling in multiproduct batch processes using matrix

Although a data processing system often works as a batch processing system, many enterprises deploy such a system as a service, which we call the service-oriented data processing system. It has been shown that in-memory data processing systems suffer from serious memory pressure. The situation becomes even worse for the service-oriented data processing systems due to various reasons. For example, in a service-oriented system, multiple submitted tasks are launched at the same time and executed in the same context in the resources, compared with the batch processing mode where the tasks are processed one by one. Therefore, the memory pressure will affect all submitted tasks, including the tasks that only incur the light memory pressure when they are run alone. In this paper, we find that the reason why memory pressure arises is because the running tasks produce massive long-living data objects in the limited memory space. Our studies further reveal that the long-living data objects are generated by the API functions that are invoked by the in-memory processing frameworks. Based on these findings, we propose a method to classify the API functions based on the memory usage rate. Further, we design a scheduler called MURS to mitigate the memory pressure. We implement MURS in Spark and conduct the experiments to evaluate the performance of MURS. The results show that when comparing to Spark, MURS can 1) decrease the execution time of the submitted jobs by up to 65.8%, 2) mitigate the memory pressure in the server by decreasing the garbage collection time by up to 81%, and 3) reduce the data spilling, and hence disk I/O, by approximately 90%.

Since the problems in bioinformatics are related to massive computing and massive data. In recent years, due to distributed computing is gaining recognition. The task originally requiring high computing power does not only rely on supercomputer. Distributed computing used off-the-shelf PC with high speed network can offer low cost and high performance computing power to handle the task. Therefore, the purpose of this paper is to implement a complete distributed computing platform based on peer-to-peer file sharing technology. The platform integrated scheduling, load balancing, file sharing, maintenance of data integrity, and user-friendly interface etc. functions. Through the platform can assist bioinformaticists in massive computing and massive data problems. Besides, the platform is easier use, more reliable, and more helpful than others for researchers to conduct bioinformatics research.

Based on the station velocity data obtained from Fujian coastal GPS network and Taiwan-Luzon GPS network and the data obtained from IGS permanent stations commonly used in the data processing of both networks, the uniformity of reference frames and velocity fields of both GPS networks is realized. It has been discovered from the analysis on the velocity field in Taiwan and its coastal areas that the horizontal crustal movements in the coastal area of Fujian, Taiwan Strait and the northern part of Taiwan Island are fully consistent. The movement direction is around 26.0°E by S and the rate is about 39 mm/a. The opposite variation occurs in the coastal mountain area in the eastern part of Taiwan Island with the movement direction of 30.0°N by W and the movement rate of about 33.3 mm/a. In the southern end of Taiwan Island, the movement direction is 50.0°S by W and the movement rate is about 13 mm/a. If the geometric center of Fujian coast is used as the reference datum, Taiwan Island has a consistent (except its north end) NW-trending movement with the direction of around 50.0°N by W. The maximum rate of 61 mm/a occurs along the eastern coast and decreases gradually towards west. The analysis of strain field indicates that Taiwan and its coastal area have a uniform strain field with the principal compressive strain direction of N48.0°W and the principal tensile strain direction of N42.0°E. The principal compressive strain rate along the eastern coast of Taiwan Island is 3.436×10−7/a, which decreases gradually towards west and reaches 1.861×10−8/a to Fujian coast. The collision and underthrust of Philippine Sea plate with Eurasian plate in the eastern part of Taiwan Island can be considered as the principal force for the crustal movement, deformation and great earthquake occurred in Taiwan and its coastal area. The direction of principal compressive stress in the area is about N55.0°W.

The gamma-ray scanning technique is widely used for evaluating the operating characteristics of distillation columns considered as the most critical components in petrochemical plants. Gamma-ray scanning provides essential data to optimize the performance of the columns and to identify maintenance requirements. New and more efficient data acquisition and processing software has been developed for Gamma-ray scanning. This new software permits to conduct gamma scanning explorations with all necessary information concerning the columns directly incorporated in the software. New features help a lot to interpret on-line the obtained results. A first version of the new data acquisition and processing software has been successfully tested in real conditions of utilization. For enhancing the interpretation of the experimental results of the technique, Geant 3 and Geant 4 Monte carlo simulation tools, developed at the CERN, have been used, to help identifying any possible malfunction inside a column and providing specific information, as a level liquid on a tray. In this study, the Geant simulations are performed for a distillation column. A set of parameters, principally the source energy ( <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">60</sup> Co) and the source-detector separation are fixed. Also many simulations were carried out to prove that there is no effect of the distillate chemical composition on the obtained results. To assure that the code runs correctly, a number of simulations were done to reproduce a density profile in the ideal case (no malfunctions in column). The distillation column process malfunctions that have been examined were principally liquid entrainment and weeping between trays. These phenomena were modelled by varying the thickness of the liquid layer on and under the tray.

Applying blockchain technology in the Internet of Things (IoT) systems to realize massive data storage is a promising technology, which can eliminate the dependence of IoT on central servers and protect data security effectively. However, in legacy IoT systems, it is hardly to store a complete blockchain at all nodes due to the limited storage space. In order to securely store massive data in the resource-constrained IoT systems, we design a blockchain-assisted distributed storage mechanism, by which multiple nodes collaboratively store a complete blockchain. Meanwhile, we design a deep reinforcement learning (DRL) based algorithm to obtain the optimal block allocation strategy for the collaborative storage. The optimal strategy can maximize the number of storage blocks so as to maximize the utilization of the limited storage space under the premise of meeting the requirements of user query delay and reliability. The simulation results show that compared with the random search algorithm and the policy gradients algorithm, the proposed Deep Q Network algorithm can achieve the maximum number of storage blocks in the same storage space, and the fastest storage speed.

The laser scanning is a technology that allows in a short time to run the relief geometric objects with a high level of detail and completeness, based on the signal emitted by the laser and the corresponding return signal. When the incident laser radiation hits the object to detect, then the radiation is reflected. The purpose is to build a three-dimensional digital model that allows to reconstruct the reality of the object and to conduct studies regarding the design, restoration and/or conservation. When the laser scanner is equipped with a digital camera, the result of the measurement process is a set of points in XYZ coordinates showing a high density and accuracy with radiometric and RGB tones. In this case, the set of measured points is called “point cloud” and allows the reconstruction of the Digital Surface Model. Even the post-processing is usually performed by closed source software, which is characterized by Copyright restricting the free use, free and open source software can increase the performance by far. Indeed, this latter can be freely used providing the possibility to display and even custom the source code. The experience started at the Faculty of Engineering in Catania is aimed at finding a valuable free and open source tool, MeshLab (Italian Software for data processing), to be compared with a reference closed source software for data processing, i.e. RapidForm. In this work, we compare the results obtained with MeshLab and Rapidform through the planning of the survey and the acquisition of the point cloud of a morphologically complex statue.

Software design for image mapping and analytics for high throughput phenotyping

We study the problem of batch learning from bandit feedback in the setting of extremely large action spaces. Learning from extreme bandit feedback is ubiquitous in recommendation systems, in which billions of decisions are made over sets consisting of millions of choices in a single day, yielding massive observational data. In these large-scale real-world applications, supervised learning frameworks such as eXtreme Multi-label Classification (XMC) are widely used despite the fact that they incur significant biases due to the mismatch between bandit feedback and supervised labels. Such biases can be mitigated by importance sampling techniques, but these techniques suffer from impractical variance when dealing with a large number of actions. In this paper, we introduce a selective importance sampling estimator (sIS) that operates in a significantly more favorable bias-variance regime. The sIS estimator is obtained by performing importance sampling on the conditional expectation of the reward with respect to a small subset of actions for each instance (a form of Rao-Blackwellization). We employ this estimator in a novel algorithmic procedure---named Policy Optimization for eXtreme Models (POXM)---for learning from bandit feedback on XMC tasks. In POXM, the selected actions for the sIS estimator are the top-p actions of the logging policy, where p is adjusted from the data and is significantly smaller than the size of the action space. We use a supervised-to-bandit conversion on three XMC datasets to benchmark our POXM method against three competing methods: BanditNet, a previously applied partial matching pruning strategy, and a supervised learning baseline. Whereas BanditNet sometimes improves marginally over the logging policy, our experiments show that POXM systematically and significantly improves over all baselines.

The mining process of traditional market equilibrium competition strategy is difficult to deal with massive data, resulting in the inability to accurately classify customer data in the process of competition strategy customization. Therefore, this paper proposes a strategy formulation method of balanced competition in the financial market based on computer data mining. Firstly, the process of the k-means clustering algorithm was optimized, and Murkowski distance and Markov distance were used as classification basis to find some potential information hidden in the data. Based on the optimized K-means clustering algorithm for data processing, in order to achieve effective data analysis, design customer behavior data mining process and analyze customer value matrix and customer pyramid. Finally, the layout framework of a balanced competition strategy in the financial market is established. The results show that the classification accuracy of the design method is higher than that of the traditional method in different states.

One of the diagnostic systems critical to the success of the DIII-D tokamak experiment is the Thomson scattering diagnostic. This diagnostic is unique in that it measures local electron temperature and density: (1) at multiple locations within the tokamak plasma; and (2) at different times throughout the plasma duration. Thomson data are digitized signals of scattered light, measured at different times and locations, from the laser beam paths fired into the plasma. Real-time acquisition of this data is performed by specialized hardware. Once obtained, the raw data are processed into meaningful temperature and density values which can be analyzed for measurement quality. This paper provides an overview of the entire Thomson scattering diagnostic software and focuses on the data acquisition, processing, and analysis software implementation. The software falls into three general categories: 1. Set-up and Control: Initializes and controls all Thomson hardware and software, sychronizes with other DIII-D computers, and invokes other Thomson software as appropriate. 2. Data Acquisition and Processing: Obtains raw measured data from memory and processes it into temperature and density values. 3. Analysis: Provides a graphical user interface in which to perform analysis and sophisticated plotting of analysis parameters.

An increasing number of reference stations have been established, leading to a sharp increase in the workload of Double-Difference (DD) baseline solutions, which are not appropriate for the integrated processing of denser networks. Correlations among the ambiguities in DD models are complex, and it is difficult to get precise solutions. This paper improves the DD ambiguity resolution performance over a long baseline, using a modified strategy based on an Un-Differenced (UD) and Un-Combined (UC) model. The satellite clocks are estimated as parameters, which are properly constrained by real-time satellite clock products for improving the smoothness of ambiguities. We use data from the Earth Scope Plate Boundary Observatory to examine the presented method in Global Positioning System (GPS) networks. Our method obtained more obviously centralised distributions. The successful fixed rate was 96·4% for the DD baseline solution, and 98·4% for the UD method. The proposed strategy is appropriate for the distributed architecture of extensive systems and avoids a heavy computational burden.

This paper describes a cyber-physical system approach for wide area control applications in power grids. Specifically, it is proposed to use cyber-physical system modeling methodology, to include explicit physical and behavior based models, to facilitate the application of traditional controller design and analysis to wide area control applications. Our proposed approach allows for modeling the physical elements of the power grid (e.g. transmission network, generation, loads and measurement devices), the cyber components (e.g. execution of reactive software algorithms in data processing and control software), and the concurrent and sequential interactions of cyber and physical components. The cyber-physical interactions are critical for wide area control applications due to the multiple spatial and temporal scales present in these systems. The modeling methodology is formulated into a classical closed loop controller which the plant and the controller contain cyber and physical elements. The proposed approach allows for studying the impacts of cyber-physical interactions, evaluating the effectiveness and resiliency of wide area controllers, and can be applied as a tool to benchmark and derive specifications for final designs. The methodology is applied to a phasor measurement unit based wide area control of a static VAR compensator in a MATLAB/Simulink environment.

GPS data processing of networks with mixed single- and dual-frequency receivers for deformation monitoring