ETL Process Design and Quality Control Research Based on Radar Spatial Data

The new generation of Doppler weather radar base data transmission is always the assessment of key project in professional work and also is the necessity for many short-term weather forecast systems. Based on the quality control of radar data, we completed the provincial or regional networking of weather radar base data, which are able to provide radar echo data on the latitude/longitude/elevation grids in real time and generate some products, so as to provide a base for the application of radar data to meso-scale weather analysis, nowcasting, and hydrological research. Simulation results show that the method is applicable and effective.

In order to use the space-time data mining technology to conduct operation research in WuLiangSuHai Eutrophication, the water quality sensor parameters of heterogeneous data which reflect the characteristics should set up a spatial data warehouse through ETL process, and water quality sensors for quality control of spatial and temporal data plays a vital role in building an effective analytical environment. The paper designs the ETL process from the data and water quality sensors artificial duty and other heterogeneous data sources spatial data, and proposes data quality control strategy based on the incremental frequency rule engine and the space the inverse distance weighting on the Combination. Experiments show that the incremental frequency rule engine could more effectively find the missing sensor data and abnormal, Space inverse distance weighting method can find the missing data and outliers in the errors within the allowed interpolation processing, ETL procedure is effective and feasible.KeywordsSensor dataFrequency increment rule engineInverse distance weighted method

Geostatistical merging of rain gauge and radar data for high temporal resolutions and various station density scenarios

Uncertainty is the intrinsic property of spatial data and one of important factors affecting the course of spatial data mining. There are diversiform forms for the essentiality and aspect of uncertainty in the spatial objects of geographic information system. Essentiality of uncertainty may consist of the components of randomicity, fuzzy, chaos, etc. And the latter, i.e. aspect of uncertainty, may include error uncertainty, location uncertainty, attribute uncertainty, topology uncertainty, inconsonance uncertainty, immaturity uncertainty and so on. Spatial data mining, which is based on uncertainty, is the course of discovering knowledge in the spatial data including the attribute of uncertainty. Spatial data is the necessary object operated by spatial data mining and its uncertainty can exist in the whole process of data input, data processing, data output, etc. Uncertainty can affect directly or indirectly the quality of spatial data mining. Specially, uncertainty of spatial data can affect directly or indirectly the veracity and reliability of ultimate decision-makings and may lead to produce false results and even reverse conclusions. Spatial data mining based on uncertainty has taken synchronously into account the two associated factors of uncertainty of data and spatial data mining. Therefore, due to considering the uncertainty, it can enhance the reliability of discovering knowledge and improve the veracity and reliability of spatial as well as raise the efficiency and practicability for the decision-makings made by operation departments. In this paper, aiming at massive spatial data belonging to the Government Geographic Information System, some work has been done under the conditions of summarizing existing relevant theory and technology of spatial data mining and uncertainty and carry out necessary practice.

Large networks of sensors are used to detect intrusions and provide security at the borders of the United States. Sensor signals are used to detect possible intrusions such as illegal immigration traffic in drugs, weapons, and smuggled goods at specific targeted geographic locations. GIS systems can be used to capture, store and analyze this location based intervention data. Using a GIS system, a spatial database can be generated from the sensor intervention data which can take into account relevant geographic information in the vicinity of the sensed interventions. Important geographic features that are close to the intervention locations such as: plateaus, hills, valleys or roadways can be extracted and added to the analysis using ArcGIS. GIS techniques alone cannot reveal meaningful hidden information within geographic data. We have developed an integrated approach involving data mining and GIS techniques to extract patterns and trends in geographic data that can aid and inform analysis. Our approach uses both spatial and association data mining techniques. Spatial data mining is the process of discovering previously unknown, interesting and potentially useful patterns from spatial datasets. Applying association rule mining to the spatial data can reveal additional important spatial relationships and help determine the relevance and importance of the sensor data. Spatial association rule mining was used to discover patterns in the intervention data, such as linking a sensed intrusion with a potentially hidden location such as a canyon, to infer a high probability of illegal traffic or immigration.

Urban flooding is one of the most serious challenges facing urban areas and causesdestruction of properties and major problems for people living in urban areas. Itoccurs in an urban area when intense convective rainfall happens that is higher thanthe capacity of the urban drainage system. Rainfall data can be recorded by radar andby rain gauge. It is essential for urban flooding studies to have access to high qualityrainfall data in attempting to reduce and tackle this challenge. The main aim of thispaper is to test the quality of radar data by means of rainfall event analysis. Radardata obtained from Yorkshire Water were analysed and compared to rain gauge dataduring selected rainfall events in a study area located in North Yorkshire in the UK.Rainfall intensity curves within 5 minute and one-hour intervals, cumulative rainfalldepth curves and normalised bias were used as standard methods for comparisonbetween radar and rain gauge data during the selected rainfall events. Available datawere collected for June, July, and October 2008 and were recorded from Blackhill,Harrogate North, and Helmsley rain gauges, and radar data for these same locationswere also collected. The results of this thesis show that certain errors might have animpact on radar data in terms of quality. Some such errors caused overestimation ofradar rainfall data, such as anomalous propagation error and error related tothunderstorm. However, other errors, such as attenuation, caused underestimation inradar rainfall data.

Tornadoes are highly destructive small-scale extreme weather processes in the troposphere. The weather radar is one of the most effective remote sensing devices for the monitoring and early warning of tornadoes. The existing tornado detection algorithms based on radar data are unsupervised and have strict multi-altitude constraints, such as the tornado detection algorithm based on tornado vortex signatures (TDA-TVS), which may lead to high false alarm rates, and the performance of the detection algorithm is greatly affected by the radar data quality control algorithm. A novel TDA-RF algorithm based on the random forest (RF) classification algorithm is proposed for real-time tornado identification of the S-band China new generation of Doppler weather radar (CINRAD-SA). The TDA-RF algorithm uses velocity features to identify tornadoes and adds features related to reflectivity and velocity spectrum width in radar level-II data. Historical CINRAD-SA tornado data from 2006–2015 are used to construct the tornado dataset and train the TDA-RF model. The performance of TDA-RF is evaluated using CINRAD-SA data from five tornadoes of 2016–2020 with enhanced Fujita(EF) scale ratings ranging from EF0 to EF4 and distances from 10 to 130 km to the radar. TDA-RF performs well overall with the probability of detection (POD), false alarm ratio (FAR), and critical success index (CSI) of 71%, 29%, and 55%, respectively. Moreover, the TDA-RF improves POD and CSI, and reduces FAR compared to the TDA-TVS. The maximum tornado early-warning time of TDA-RF is 17 min, and the average is 6 min; TDA-RF can provide classification probability according to the tornado generation and development process to facilitate tracking ability.

With tremendous development in the field of science and technology, there is vast amount of data which are used in analytics for decision making. Considering its spatial characteristics for mining will enhance the accuracy of decision. So, obtaining knowledge from spatial data becomes very essential and meaningful. The spatial database contains very numerous amounts of spatial and non-spatial data of different forms. Interpretation and analyzing of vast data is far beyond human ability. In order to acquire knowledge on such scenario we need spatial data mining. The challenges involved in spatial mining are to deal with different objects that represent the spatial characteristics. This makes spatial data mining a dominant research field. This chapter briefs about the characteristics of spatial data mining and the methods of spatial data mining in recent years.

Uncertainties pervade spatial data mining. This paper proposes a method of spatial data mining handling randomness and fuzziness simultaneously. First, the uncertainties in spatial data mining are presented via characteristics, spatial data, knowledge discovery and knowledge representation. Second, the aspects of the uncertainties in spatial data mining are briefed. They often appear simultaneously, but most of the existing methods cannot deal with spatial data mining with more than one uncertainty. Third, cloud model is presented to mine spatial data with both randomness and fuzziness. It may also act as an uncertainty transition between a qualitative concept and its quantitative data, which is the basis of spatial data mining in the contexts of uncertainties. Finally, a case study on landslide-monitoring data mining is given. The results show that the proposed method can well deal with randomness and fuzziness during the process of spatial data mining.

Extraction of interesting knowledge from large spatial databases is an important task in the development of spatial database systems. Spatial data mining is the branch of data mining that deals with spatial (location) data. Analyzing the huge amount (usually tera-bytes) of spatial data obtained from large databases such as credit card payments, telephone calls, environmental records, census demographics etc. is, however, a very difficult task. Visual data mining applies human visual perception to the exploration of large data sets. Presenting data in an interactive, graphical form often fosters new insights, encouraging the formation and validation of new hypotheses to the end of better problem-solving and gaining deeper domain knowledge. In this paper we give a short overview of visual data mining techniques, especially the area of analyzing spatial data. We provide some examples for effective visualizations of spatial data in important application areas such as consumer analysis, e-mail traffic analysis, and census demographics.

Spatial data mining and spatial data visualization are two comparatively popular technical methods in recent years, in essence, both purpose is to find geography phenomena what spatial data express and find various knowledge and laws implicit in geography entity. so it is necessary to combine both organically and form a new research direction - Visualization Spatial Data Mining (VSDM). This paper mainly discusses the key relationships of visualization and spatial data mining, the main Application of visualization theories and technologies in spatial data mining, the main methods and examples of visualization spatial data mining, we also present a reference model Visualization Spatial Data.

Quality control procedures based on nonvelocity parameters for use with a short-range radar system are applied with slight modification to long-range radar data collected offshore of North Carolina. The radar footprint covers shelf and slope environments and includes a segment of the Gulf Stream (GS). Standard processed and quality controlled (QCD) radar data are compared with 4 months of acoustic Doppler current profiler (ADCP) time series collected at three different sites within the radar footprint. Two of the ADCP records are from the shelf and the third is on the upper slope and is frequently within the GS. Linear regression and Bland–Altman diagrams are used to quantify the comparison. QCD data at all sites have reduced scatter and improved correlation with ADCP observations relative to standard processed data. Uncertainty is reduced by approximately 20%, and linear regression slopes and correlation coefficients increase by about 0.1. At the upper slope site, the QCD data also produced a significant increase in the mean speed. Additionally, a significant increase, averaging roughly 20%, in mean speed in the GS is apparent when comparing standard processed data and QCD data, concentrated at large range and at the azimuthal extremes of radial site coverage. Shifts in the distributions of the standard processed and QCD velocity estimates are consistent with the removal of zero-mean noise from the observations, which has minimal impact where the radial site range is <70 km and a large impact at greater range in the GS where mean currents exceed 1 m s−1.

Spatial Data Warehouses (SDW) and Spatial OLAP (SOLAP) systems are well-known Business Intelligence technologies that aim to support multidimensional and online analysis of huge volumes of datasets with spatial reference. Spatial vagueness is one of the most neglected imperfections of spatial data. Although several works propose new ad-hoc models for handling spatial vagueness in information systems, the implementation of those models in Spatial DBMS and SDW is still in an embryonic state. Thus, in this paper, we present a new design method for SOLAP datacubes that allows handling vague spatial data analysis issues. This method relies on a risk management method applied to the potential risks of data misinterpretation and decision-makers’ tolerance levels to those risks. We also present a system implementing our method.

Spatial data refers to extracting and mining the hidden, implicit, valid, novel and interesting spatial or non-spatial patterns or rules from large-amount, incomplete, noisy, fuzzy, random, and practical spatial databases. In which an important issue but remains underdeveloped is to reveal and handle the uncertainties in spatial data mining. In This work, uncertainty of spatial data is briefly analyzed firstly, including the types and origins of uncertainty, their models of measurement and propagation. Then, some uncertainty factors in operation of spatial data are discussed and some uncertainty handling methods are adopted, including maximum variance data discretization and fuzzy belief function. Finally, we think the process of spatial data can be regarded as a complex system, a linear serial processing system in engineering control systems. An uncertainty propagation model of spatial data - fuzzy logic uncertainty propagation model with credibility factor is developed. Moreover, several key problems about uncertainty handling and propagation in spatial data are put forward.

This paper describes a new velocity–azimuth display (VAD)-based dealiasing method developed for automated radar radial velocity data quality control to satisfy the high-quality standard and efficiency required by operational radar data assimilation. The method is built on an alias-robust velocity–azimuth display (AR-VAD) analysis. It upgrades and simplifies the previous three-step dealiasing method in three major aspects. First, the AR-VAD is used with sufficiently stringent threshold conditions in place of the original modified VAD for the preliminary reference check to produce alias-free seed data in the first step. Second, the AR-VAD is more accurate than the traditional VAD for the refined reference check in the original second step, so the original second step becomes unnecessary and is removed. Third, a block-to-point continuity check procedure is developed, in place of the point-to-point continuity check in the original third step, which serves to enhance the use of the available seed data in a properly enlarged block area around each flagged data point that is being checked with multiple threshold conditions to avoid false dealiasing. The new method has been tested extensively with aliased radial velocity data collected under various weather conditions, including hurricane high-wind conditions. The robustness of the new method is exemplified by the results tested with three cases. The limitations of the new method and possible improvements are discussed.