ON VOLUME DATA REDUCTION FOR LIDAR DATASETS

A test pattern compression scheme for test data volume and application time reduction is proposed. While compression reduces test data volume, the increased number of internal scan chains due to an on-chip, fixed-rate decompressor reduces test application time proportionately. Through on-chip decompression, both the number of virtual scan chains visible to the ATE and the functionality of the ATE are retained intact. Complete fault coverage is guaranteed by constructing the decompression hardware deterministically through analysis of the test pattern set.

The Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard the Terra satellite was designed to generate along‐track stereo images. The data are available at low cost, providing a feasible opportunity for generating digital elevation models (DEMs) in areas where little or no elevation data are yet available. This study evaluates the accuracy of DEMs extracted from ASTER data covering mountainous terrain. For an assessment of the achieved accuracies in the Andean study site, comparisons were made to similar topographical conditions in Switzerland, where reference data were available. All raw DEMs were filtered and interpolated by the post‐processing tools included with PCI Geomatica, the software package used. After carefully checking the DEM quality, further post‐processing was undertaken to eliminate obvious artefacts such as peaks and sinks. Accuracy was tested by comparing the DEMs in the Swiss Alps to three reference models. The achieved results of the generated DEMs are promising, considering the extreme terrain. Given accurate and well‐distributed ground control points (GCPs), it is possible to generate DEMs with a root mean square (RMS) error between 15 m and 20 m in hilly terrain and about 30 m in mountainous terrain. The DEMs are very accurate in nearly flat regions and on smooth slopes with southern expositions: errors are generally within ±10 m in those cases. Larger errors do appear in forested, snow covered or shady areas and at steep cliffs and deep valleys with extreme errors of a few hundred metres. The evaluation showed that the quality of the DEMs is sufficient for enabling atmospheric, topographic and geometric correction to various satellite datasets and for deriving additional products.

Abstract. Digital Elevation Models (DEMs) are considered as one of the most relevant geospatial data to carry out land-cover and land-use classification. This work deals with the application of a mathematical framework based on a Gaussian Markov Random Field (GMRF) to interpolate grid DEMs from scattered elevation data. The performance of the GMRF interpolation model was tested on a set of LiDAR data (0.87 points/m2) provided by the Spanish Government (PNOA Programme) over a complex working area mainly covered by greenhouses in Almería, Spain. The original LiDAR data was decimated by randomly removing different fractions of the original points (from 10% to up to 99% of points removed). In every case, the remaining points (scattered observed points) were used to obtain a 1 m grid spacing GMRF-interpolated Digital Surface Model (DSM) whose accuracy was assessed by means of the set of previously extracted checkpoints. The GMRF accuracy results were compared with those provided by the widely known Triangulation with Linear Interpolation (TLI). Finally, the GMRF method was applied to a real-world case consisting of filling the LiDAR-derived DSM gaps after manually filtering out non-ground points to obtain a Digital Terrain Model (DTM). Regarding accuracy, both GMRF and TLI produced visually pleasing and similar results in terms of vertical accuracy. As an added bonus, the GMRF mathematical framework makes possible to both retrieve the estimated uncertainty for every interpolated elevation point (the DEM uncertainty) and include break lines or terrain discontinuities between adjacent cells to produce higher quality DTMs.

Abstract. In synthetic aperture radar (SAR) technology, urban mapping and modelling have become possible with revolutionary missions TerraSAR-X (TSX) and Cosmo-SkyMed (CSK) since 2007. These satellites offer 1m spatial resolution in high-resolution spotlight imaging mode and capable for high quality digital surface model (DSM) acquisition for urban areas utilizing interferometric SAR (InSAR) technology. With the advantage of independent generation from seasonal weather conditions, TSX and CSK DSMs are much in demand by scientific users. The performance of SAR DSMs is influenced by the distortions such as layover, foreshortening, shadow and double-bounce depend up on imaging geometry. In this study, the potential of DSMs derived from convenient 1m high-resolution spotlight (HS) InSAR pairs of CSK and TSX is validated by model-to-model absolute and relative accuracy estimations in an urban area. For the verification, an airborne laser scanning (ALS) DSM of the study area was used as the reference model. Results demonstrated that TSX and CSK urban DSMs are compatible in open, built-up and forest land forms with the absolute accuracy of 8–10 m. The relative accuracies based on the coherence of neighbouring pixels are superior to absolute accuracies both for CSK and TSX.

Slope angle and aspect as influencing factors on the accuracy of the SRTM and the ASTER GDEM databases

Today’s scientific simulations generate exceptionally large volumes of data, challenging the capacities of available I/O bandwidth and storage space. This necessitates a substantial reduction in data volume, for which error-bounded lossy compression has emerged as a highly effective strategy. A crucial metric for assessing the efficacy of lossy compression is visualization. Despite extensive research on the impact of compression on visualization, there is a notable gap in the literature concerning the effects of compression on the visualization of Adaptive Mesh Refinement (AMR) data. AMR has proven to be a potent solution for addressing the rising computational intensity and the explosive growth in data volume that requires storage and transmission. However, the hierarchical and multi-resolution characteristics of AMR data introduce unique challenges to its visualization, and these challenges are further compounded when data compression comes into play. This article delves into the intricacies of how data compression influences and introduces novel challenges to the visualization of AMR data.

Digital elevation models (DEMs) are fundamental data infrastructure that support a large range of applications in environmental modelling. The distribution of solar radiation, patterns of wetness and water movement, mobility of animals and evolution of soils are all intimately linked to topographic form. DEMs for the Australian continent have progressed from 1/10 th degree resolution in 1982 to 1/40 th degree in 1991, 9 th degree) in 1996 and now 1 second (about 30 m) derived from the NASA Shuttle Radar Topographic Mission (SRTM). Each improvement in resolution and accuracy enables new applications as more detail and structure are revealed. The 1 second SRTM-derived DEMs for Australia are now publicly available through the Geoscience Australia National Elevation Data Framework (NEDF) portal (http://nedf.ga.gov.au). The raw SRTM data has been processed through a series of steps to provide several products suitable for different purposes: • DEM-S is a bare-earth elevation model with stripes and voids removed, offsets due to trees treated and noise substantially reduced in flatter areas by adaptive smoothing, which is suitable for most applications where connectivity of flow is not the primary concern such as calculation of slopes and topographic position • DEM-H is a hydrologically conditioned elevation model derived from DEM-S with drainage enforcement using 1:250,000 scale stream lines, which is suitable for applications where hydrological connectivity is of primary concern such as calculation of contributing areas and catchment delineation • DEM is the pre-cursor to DEM-S before adaptive smoothing, which may be useful when the retention of detail is more important than smoothness. There is also a cleaned digital surface model (DSM) that is the precursor to DEM-S before tree offsets are removed, which is restricted to Australian government use only. Most users of DEMs need information about the shape of the landscape, not just height, so we are producing a set of standard derived products including slope, aspect, curvatures, topographic position, relief, contributing area and wetness index. These derived surfaces are key inputs for continental scale prediction of soil properties and ecological patterns. Predictions of a suite of key soil properties on a 3 second grid for the Australian continent, using the 1 second terrain layers, are being developed and will contribute to the GlobalSoilMap.net global grid of soil properties. Catchments and stream lines derived from DEM-H will form the basis for future versions of the Australian Hydrological Geospatial Fabric (http://www.bom.gov.au/water/geofabric/index.shtml). Digital elevation data continues to improve as measurement technology improves. Airborne lidar already provides high precision terrain data with vertical accuracy of 20 cm or better at 1 m horizontal resolution or better. This data is available in parts of Australia where the better accuracy justifies the high cost, such as urban areas, coastal lowlands, wetlands and floodplains. We need to develop ways of using high resolution data covering limited areas in combination with lower resolution data covering the continent. New global DEMs are being acquired and will replace SRTM in the years ahead. The TANDEM-X satellite radar system will potentially produce global elevations at 2-3 times better resolution and accuracy than SRTM, and other systems will follow. Higher resolutions bring further technical challenges, not least in data volume: each 1 second DEM product for Australia is a 40 GB dataset and finer resolutions will be correspondingly larger.

Reducing data volume in big data: Parallel processing based data filtering techniques Professor Shikharesh Majumdar from Carleton University examines reducing data volume in big data, focusing on parallel processing based data filtering techniques. Volume, velocity, and variety are three well-known characteristics of big data. The large data volumes often introduce formidable challenges to processing such data in a timely and economical manner. Research on data filtering is underway under the leadership of Shikharesh Majumdar at the Real Time and Distributed Systems Research Centre at Carleton University. Users are often interested only in a subset of the raw data.

Question: Predictive vegetation modelling relies on the use of environmental variables, which are usually derived from abase data set with some level of error, and this error is propagated to any subsequently derived environmental variables. The question for this study is: What is the level of error and uncertainty in environmental variables based on the error propagated from a Digital Elevation Model (DEM) and how does it vary for both direct and indirect variables?Location: Kioloa region, New South Wales, AustraliaMethods: The level of error in a DEM is assessed and used to develop an error model for analysing error propagation to derived environmental variables. We tested both indirect (elevation, slope, aspect, topographic position) and direct (average air temperature, net solar radiation, and topographic wetness index) variables for their robustness to propagated error from the DEM.Results: It is shown that the direct environmental variable net solar radiation is less affected by error in the DEM than the indirect variables aspect and slope, but that regional conditions such as slope steepness and cloudiness can influence this outcome. However, the indirect environmental variable topographic position was less affected by error in the DEM than topographic wetness index. Interestingly, the results disagreed with the current assumption that indirect variables are necessarily less sensitive to propagated error because they are less derived.Conclusions: The results indicate that variables exhibit both systematic bias and instability under uncertainty. There is a clear need to consider the sensitivity of variables to error in their base data sets in addition to the question of whether to use direct or indirect variables.

The DEAP-3600 experiment will search for dark matter particle interactions on liquid Argon at SNOLAB, located 2 km underground in Sudbury Ontario. The DEAP-3600 detector is currently under construction and will be instrumented with 255 Hamamatsu PMT, each channel being decoupled from the PMT high voltage by a set of custom analogue signal conditioning boards. The signal conditioning boards split the PMT signals into three outputs: high gain; low gain; and into a twelve channel analogue sum (ASUM). Three different digitisers sampling at 250 MS/s, 62.5 MS/s and 50 MS/s digitise the signal conditioning outputs, the 50 MS/s sampler being a custom 6U VME Stratix IV FPGA based module generating trigger decisions based on a pulse-shape discrimination (PSD) selection. The 250 MS/s and 62.5 MS/s digitisers are commercial CAEN V1720 and V1740 modules respectively. DAQ from the digitisers to disk is handled by the Midas system with additional software event filtering provided by DEAP-3600 specific algorithms. With a total mass of 3600 kg the expected Argon β decay background is 3.6 kHz, a key feature of the DEAP-3600 TDAQ system is the reduction in data volume from 4500 MB/s to 5 MB/s, while recording all events of interest.

Advanced algorithmic solutions are necessary to process the ever-increasing amounts of mass spectrometry data that are being generated. In this study, we describe the falcon spectrum clustering tool for efficient clustering of millions of MS/MS spectra. falcon succeeds in efficiently clustering large amounts of mass spectral data using advanced techniques for fast spectrum similarity searching. First, high-resolution spectra are binned and converted to low-dimensional vectors using feature hashing. Next, the spectrum vectors are used to construct nearest neighbor indexes for fast similarity searching. The nearest neighbor indexes are used to efficiently compute a sparse pairwise distance matrix without having to exhaustively perform all pairwise spectrum comparisons within the relevant precursor mass tolerance. Finally, density-based clustering is performed to group similar spectra into clusters. Several state-of-the-art spectrum clustering tools were evaluated using a large draft human proteome data set consisting of 25 million spectra, indicating that alternative tools produce clustering results with different characteristics. Notably, falcon generates larger highly pure clusters than alternative tools, leading to a larger reduction in data volume without the loss of relevant information for more efficient downstream processing. falcon is a highly efficient spectrum clustering tool, which is publicly available as an open source under the permissive BSD license at https://github.com/bittremieux/falcon.

3D single-photon LiDAR imaging has an important role in many applications. However, full deployment of this modality will require the analysis of low signal to noise ratio target returns and very high volume of data. This is particularly evident when imaging through obscurants or in high ambient background light conditions. This paper proposes a multiscale approach for 3D surface detection from the photon timing histogram to permit a significant reduction in data volume. The resulting surfaces are background-free and can be used to infer depth and reflectivity information about the target. We demonstrate this by proposing a hierarchical Bayesian model for 3D reconstruction and spectral classification of multispectral single-photon LiDAR data. The reconstruction method promotes spatial correlation between point-cloud estimates and uses a coordinate gradient descent algorithm for parameter estimation. Results on simulated and real data show the benefits of the proposed target detection and reconstruction approaches when compared to state-of-the-art processing algorithms.

Despite its promise, RFID technology presents numerous challenges, including incomplete data, lack of location and containment information, and very high volumes. In this work, we present a novel data interpretation and compression substrate over RFID streams to address these challenges in enterprise supply-chain environments. Our results show that our inference techniques provide good accuracy while retaining efficiency, and our compression algorithm yields significant reduction in data volume.

Various compression schemes have been suggested for storage and distribution of hyperspectral remotely sensed data. Hyperspectral forestry applications that rely on the measurement of subtle variations in the spectral signature of the forest canopy can be affected by modification of the spectra induced by compression. As part of an experiment for the Canadian Space Agency (CSA), Hyperion data cubes acquired over the Greater Victoria Watershed District (GVWD) were compressed using the Successive Approximation Multi-stage Vector Quantization (SAMVQ) and Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ) algorithms developed by CSA. The data were compressed using compression ratios 10:1 and 20:1 and were returned uncompressed. The data cubes were classified into forest species using the same supervised classification methodology as applied to the original data. The classification accuracies were compared. For some applications, one can achieve significant reductions in data volume through compression. Of the compression algorithms and ratios tested, SAMVQ 10:1 has the least overall effect but still reduces classification accuracies on difficult to separate classes. While uncompressed data are preferred, SAMVQ 10:1 compression may be suitable for forest inventory.

Using computer programs to simulate protein complex formation is an important approach for better comprehension of the interaction mechanisms of such molecules. The parametric representation and the Zernike polynomial method are both compact representations of molecular surfaces based on spherical harmonics, used for visualization and comparison of molecules and for protein-protein docking. They pose, however, limitations regarding surface topology and difficulty in representing arbitrary functions. In this study, the representation capacity of such methods were refined to attain high quality reproductions and applicability to arbitrary topologies. Throughout the analysis of several algorithms, the stages of surface mesh calculation and spherical mapping were identified as highly influential on the quality of the spherical harmonics parametric representation, while high sensibility to changes in the function values were identified as an influential factor for projections in 3D Zernike. A surface calculation method that generates a highly regular mesh was adopted and paired with a modern spherical mapping algorithm to yield reconstructions with low level of distortions and high quality surfaces. Similarity among surfaces from different structures in a conformational ensemble were detected to allow sharing of portions of the description among several surfaces, with corresponding reduction in data volume and processing demand and controllable influence of distortions. A new input data organization method improved the reconstruction quality of general functions in 3D Zernike, although introducing a map to restore the function. Results indicate promising application of the methods in highly detailed protein-protein docking.