Visualisation of network forensics traffic data with a self-organising map for qualitative features

<b><sc>Abstract.</sc></b> The area irrigated by furrow irrigation in the U.S. has been steadily decreasing but still represents about 20% of the total irrigated area in the U.S. Furrow irrigation sediment loss is a major water quality issue in the western U.S. and a method for estimating sediment loss is needed to quantify the environmental impacts and estimate effectiveness and economic value of conservation practices. The objective of the study was to investigate the use of the unsupervised machine learning technique Kohonen self-organizing maps (KSOM) to predict furrow sediment loss. Historical published and unpublished data sets containing measurements of furrow irrigation sediment loss in the western U.S. were assembled into a furrow sediment loss data set comprising over 2000 measurements. Despite the immunity of KSOMs to measurement variability, the inherent variability in measured furrow sediment loss limited the ability of a KSOM model to reliability predict furrow sediment loss. Furrow sediment loss was under predicted by 44% on average with a linear regression coefficient of determination of 0.6. The KSOM model was placing little weight on measured sediment loss in the input data set, indicating that it was clustering the data based on input parameters defining the hydraulic and soil conditions. This outcome was used to develop a transfer learning approach for predicting furrow sediment loss. The transfer learning approach used a KSOM to cluster data records of similar hydraulic and soil conditions in the data set. Mean measured sediment loss and furrow flow rate of each cluster was determined based on data set vectors assigned to a cluster by the KSOM. Furrow sediment loss prediction was obtained by applying an input vector to the KSOM to identify the cluster the input vector most closely matches. Then the mean measured sediment loss of the identified cluster was adjusted for any difference between the input vector furrow flow rate and cluster mean furrow flow rate to obtain a prediction of furrow sediment loss. Predicted furrow sediment loss was 16% less than measured sediment loss on average with a coefficient of determination of 0.82. When the data set was randomly split into model development (90%) and validation (10%) data sets the prediction results were similar.

Navigation in Databases Using Self-Organising Maps

A parallel low rank matrix optimization method for recovering internet traffic network data via link flow measurement

This paper discusses the use of self-organizing map (SOM) in the image coding. The image is partitioned into blocks each of which is fed to the SOM as the training vectors. Then they are quantized into smaller number of vectors that is used as codeword to reconstruct the image. Thus the data size is reduced. In reconstructing the image, each block is filled with the corresponding codeword. The feasibility of the system is tested by computer simulations and the effect of the sizes of blocks and SOM on the compression rate and image quality is studied. Then the SOM based hardware image coding system is designed, and its functionality is also tested. It is estimated that the hardware SOM can code a single image within 1.2 ms.

Self-organizing maps based on limit cycle attractors

A powerful method in knowledge discovery and cluster extraction is the use of self-organizing maps (SOMs), which provide adaptive quantization of the data together with its topologically ordered lower-dimensional representation on a rigid lattice. The knowledge extraction from SOMs is often performed interactively from informative visualizations. Even though interactive cluster extraction is successful, it is often time consuming and usually not straightforward for inexperienced users. In order to cope with the need of fast and accurate analysis of increasing amount of data, automated methods for SOM clustering have been popular. In this study, we use spectral clustering, a graph partitioning method based on eigenvector decomposition, for automated clustering of the SOM. Experimental results based on seven real data sets indicate that spectral clustering can successfully be used as an automated SOM segmentation tool, and it outperforms hierarchical clustering methods with distance based similarity measures.KeywordsSpectral ClusterHierarchical Agglomerative ClusterEigenvector DecompositionHierarchical Agglomerative Cluster MethodCluster ExtractionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A powerful method in analysis of large data sets where there are many natural clusters with varying statistics such as different sizes, shapes, density distribution, is the use of sel-forganizing maps (SOMs) [1]. However, further processing tools, such as visualization, interactive clustering, are often necessary to capture the clusters from the learned SOM knowledge. A recent visualization scheme, CONNvis [2], and interactive clustering from CONNvis, utilizes the data topology for SOM knowledge representation by using a weighted Delaunay graph, CONN. In this paper, an automated clustering scheme for SOMs, SOMcluster, which is a two-level clustering of CONN by the skills obtained in the interactive process, is proposed. It is shown that SOMcluster, which does not require the number of clusters a priori, is used successfully for automated segmentation of a remote sensing spectral image which has many clusters some of which were unidentified in previous works.

Privacy preserving data mining of sequential patterns for network traffic data

Use of self-organizing maps in modelling the distribution patterns of gammarids (Crustacea: Amphipoda)

The combined use of self-organizing map technique and fuzzy c-means clustering to evaluate urban groundwater quality in Seoul metropolitan city, South Korea

In this study, the method of self-organizing maps (SOMs) is used with NCEP–NCAR reanalysis data to advance the continuum perspective of Northern Hemisphere teleconnection patterns and to shed light on the secular eastward shift of the North Atlantic Oscillation (NAO) that began in the late 1970s. A 20-pattern SOM analysis of daily, wintertime, Northern Hemisphere sea level pressure reveals a continuum of patterns that correspond closely with well-known teleconnection patterns. This analysis also reveals that interdecadal variability of the hemispheric sea level pressure field may be understood in terms of changes in the frequency distribution within the continuum of sea level pressure patterns described by the SOM. Based on the continuum perspective illustrated with the SOM, the above secular shift of the NAO may be understood as a change in dominance from westward-displaced, negative NAO-like patterns to eastward-displaced, positive NAO-like patterns, though westward- and eastward-displaced NAO-like patterns existed during all time periods and for both phases.

The paper deals with evaluation of automatic training samples selection method based on self-organizing map (SOM) in face recognition systems. In earlier paper [1] we presented an approach for automatic training samples selection using various clustering algorithms with good results on the CMU PIE face database. We showed that with the use of SOM we can achieve a good training samples selection. In this paper we further evaluate this approach with the use of face recognition systems based on principal component analysis (PCA) and support vector machines (SVM). We compare the results with random (uncontrolled and controlled) training samples selection and we evaluate the recognition accuracy of each method.

The loss of traffic state data is a common problem in intelligent transportation system. To improve the imputation accuracy and robustness of road traffic data, a novel generative adversarial network for the imputation of road network traffic data is proposed in this paper, i.e., GAE-GAN-LSTM. The spatiotemporal characteristics were extracted using a improved graph auto-encoder (GAE), followed by a generative adversarial network (GAN) to generate the complete spatiotemporal characteristics on the basis of the missing features. The internal structure of the generator was a long short-term memory network (LSTM), and the internal structure of the discriminator was a fully connected neural network (FCN). Finally, the traffic state data could be recovered by the decoder of GAE. The experimental results revealed that the performance of the proposed method was better than that of the other methods at any data loss ratio considered. The main innovations of the proposed method include two aspects. One, an improved GAE for the imputation of road network data was presented by redefining the loss function of GAE, which could effectively extract the potential spatiotemporal features of a road network. Two, the GAN was used to generate the spatiotemporal characteristics of the traffic state data by using the strong data generation ability of GAN.KeywordsData imputationGraph auto-encoderGenerative adversarial networkRoad traffic state data

Data missing remains a difficult and important problem in the transportation information system, which seriously restricts the application of the intelligent transportation system (ITS), dominatingly on traffic monitoring, e.g., traffic data collection, traffic state estimation, and traffic control. Numerous traffic data imputation methods had been proposed in the last decade. However, lacking of sufficient temporal variation characteristic analysis as well as spatial correlation measurements leads to limited completion precision, and poses a major challenge for an ITS. Leveraging the low-rank nature and the spatial-temporal correlation of traffic network data, this paper proposes a novel approach to reconstruct the missing traffic data based on low-rank matrix factorization, which elaborates the potential implications of the traffic matrix by decomposed factor matrices. To further exploit the temporal evolvement characteristics and the spatial similarity of road links, we design a time-series constraint and an adaptive Laplacian regularization spatial constraint to explore the local relationship with road links. The experimental results on six real-world traffic data sets show that our approach outperforms the other methods and can successfully reconstruct the road traffic data precisely for various structural loss modes.

A powerful method in the analysis of datasets where there are many natural clusters with varying statistics such as different sizes, shapes, density distribution, overlaps, etc., is the use of self-organizing maps (SOMs). However, further processing tools, such as visualization and interactive clustering, are often necessary to capture the clusters from the learned SOM knowledge. A recent visualization scheme (CONNvis) and its interactive clustering utilize the data topology for SOM knowledge representation by using a connectivity matrix (a weighted Delaunay graph), CONN. In this paper, we propose an automated clustering method for SOMs, which is a hierarchical agglomerative clustering of CONN. We determine the number of clusters either by using cluster validity indices or by prior knowledge on the datasets. We show that, for the datasets used in this paper, data-topology-based hierarchical clustering can produce better partitioning than hierarchical clustering based solely on distance information.