An intelligent, interactive tool for exploration and visualization of time-oriented security data

The objective of this paper is to propose a model of a distributed intrusion detection system based on the multi-agent paradigm and the distributed file system (HDFS). Multi-agent systems (MAS) are very suitable to intrusion detection systems as they can address the issue of geographic data security in terms of autonomy, distribution and performance. The proposed system is based on a set of autonomous agents that cooperate and collaborate with each other to effectively detect intrusions and suspicious activities that may impact geographic information systems. Our system allows the detection of known and unknown computer attacks without any human intervention (Security Experts) unlike traditional intrusion detection systems that rely on knowledge bases as a mechanism to detect known attacks. The proposed model allows a real time detection of known and unknown attacks within large networks hosting geographic data.

With the fast development of industrial Internet of things (IIoT), a large amount of data is being generated continuously by different sources. Storing all the raw data in the IIoT devices locally is unwise considering that the end devices’ energy and storage spaces are strictly limited. In addition, the devices are unreliable and vulnerable to many threats because the networks may be deployed in remote and unattended areas. In this paper, we discuss the emerging challenges in the aspects of data processing, secure data storage, efficient data retrieval and dynamic data collection in IIoT. Then, we design a flexible and economical framework to solve the problems above by integrating the fog computing and cloud computing. Based on the time latency requirements, the collected data are processed and stored by the edge server or the cloud server. Specifically, all the raw data are first preprocessed by the edge server and then the time-sensitive data (e.g., control information) are used and stored locally. The non-time-sensitive data (e.g., monitored data) are transmitted to the cloud server to support data retrieval and mining in the future. A series of experiments and simulation are conducted to evaluate the performance of our scheme. The results illustrate that the proposed framework can greatly improve the efficiency and security of data storage and retrieval in IIoT.

We use Apache Spark analytics to investigate the logs of an operational cloud object store service to understand how it is being used. This investigation involves going over very large amounts of historical data (PBs of records in some cases) collected over long periods of time retroactively. Existing tools, such as Elasticsearch-Logstash-Kibana (ELK), are mainly used for presenting short-term metrics and can-not perform advanced analytics such as machine learning. A possible solution is to save for long periods only certain aggregations or calculations produced from the raw log data, such as averages or histograms, however these must be decided in advance, and cannot be changed retroactively since the raw data has already been discarded. Spark allows us to gain insights going over historical data collected over long periods of time and to apply the historical models on online data in a simple and efficient way.

Advances in information technology enable a capability to gather large amounts of raw data from all parts of our society to a degree of efficiency that can actually encumber surveillance missions. Today one can create automated networks of sensors to gather enormous volumes of data, but we do not have the human capacity to sort through all this raw data. Similar problems exist in the area of machinery condition or health monitoring - another surveillance problem. By automating data collection, processing, fusion and interpretation, one can bring the most relevant and timely information to human analysts, planners, and responders. Key technologies that enable this transformation from data to knowledge are distributed hardware and software architectures, intelligent sensors, data fusion and reasoning algorithms, and open system architectures for information exchange. Distributed hardware and software structures partition complex systems into a collection of interconnected subsystems. Intelligent sensors enable this conversion of data to information by processing massive amounts of data at the subsystem and higher levels to extract contextually relevant information. The transformation from raw sensor data to useful information requires the application of subsystem-specific signal processing and feature extraction algorithms, data fusion, and classification algorithms to combine data and features from commensurate and non-commensurate sensors or information sources. Emerging standards in open system architectures for condition based maintenance apply equally to surveillance systems and condition monitoring systems. Examples from fielded system health monitoring applications are presented along with their parallels to surveillance systems with application to homeland defense.

Training task in classical machine learning models, such as deep neural networks, is generally implemented at a remote cloud center for centralized learning, which is typically time-consuming and resource-hungry. It also incurs serious privacy issue and long communication latency since a large amount of data are transmitted to the centralized node. To overcome these shortcomings, we consider a newly-emerged framework, namely federated edge learning, to aggregate local learning updates at the network edge in lieu of users' raw data. Aiming at accelerating the training process, we first define a novel performance evaluation criterion, called learning efficiency. We then formulate a training acceleration optimization problem in the CPU scenario, where each user device is equipped with CPU. The closed-form expressions for joint batchsize selection and communication resource allocation are developed and some insightful results are highlighted. Further, we extend our learning framework to the GPU scenario. The optimal solution in this scenario is manifested to have the similar structure as that of the CPU scenario, recommending that our proposed algorithm is applicable in more general systems. Finally, extensive experiments validate the theoretical analysis and demonstrate that the proposed algorithm can reduce the training time and improve the learning accuracy simultaneously.

Data visualization is basically a field that deals with graphical representation of information from raw data. Data visualization tools provide areachable way to see and understand the pattern, outliers and trends in data by using data tools like charts, graphs and maps. It makes decision based on knowing rather than guessing. Large amount of data can be accessible because of Data visualization. Data Visualization also helps in telling stories that highlight trends, pattern, and outliers in organization. Thispaper basically focuses on how data visualization is important in sector of education and also the role of dashboard in improvement of students performance and institutional growth.

With the evolving of high-throughput experimental techniques, large amounts of molecular profiling data are becoming available for regular clinical studies. These data need to be stored, processed, archived, distributed and, more importantly, linked. In ELIXIR pilot project, we focus on connecting the archival storage of such data, with databases that store the processed data and visualised workflow systems that manage the computational pipelines. After perusing the processed data, users often come back to the raw data not only to reconfirm the data processing but also to further explore the raw data in workflow systems like Galaxy. In the Translational Research IT (TraIT) project of Center for Translational Molecular Medicine, raw data are stored in European Genome-phenome Archive (EGA); and processed data, tranSMART. In light of the distinct interpretations of data ontology structures in different repositories, we are aiming to establish the a systematic, flexible and sustainable mapping between the processed data and the corresponding raw data regardless of their repositories. We propose an ontology structure with a few stable mapping concepts — Study, Sample Collected Experimental Data (SCED) and File — to connect different repositories, separating the three essential processes: data uploading, data ontology structuring and data retrieving, in such a way that a general upload and structuring methodology can be used for malleable data retrieving. To demonstrate this ontology structure, we create a RDF graph demo using CTMM-TraIT Cell Line Use Case (CLUC) data.

Large amount of data in recent years have pushed experts to use data mining techniques in all fields. Data mining is a process in which useful information from raw data is obtained; it can also be used in classification problems. Lately, in some systems, especially in medical systems, experts have tried to combine data mining techniques and evolutionary algorithms to get accurate results. One of the most critical diseases, which has a considerable mortality rate in the world, is coronary artery disease. To improve the diagnosis of this dangerous disease in the early stages, we proposed a system which uses data mining techniques and an evolutionary algorithm called Imperialist Competitive algorithm (ICA). The proposed system used an algorithm based on the decision tree to reduce the data dimension and to produce valid rules. Then, a fuzzy system is created. Tuning fuzzy membership functions were done using ICA and Improved ICA to optimize the results. Since the convergence speed is one of the important factors in an evolutionary algorithm, a change was made in this algorithm so that the convergence occurs more quickly. The results show that ICA and Improved ICA produce the same results in classification accuracy, but the convergence time is different. The proposed system gets an accuracy of 94.92 %, which is high in comparison with similar works.

An integrated federated learning algorithm for short-term load forecasting

Genomic prediction is now a mature technology for animal breeding and genetics and many breeders’ associations and livestock, and poultry genetics companies have adopted high-through genotyping for routine breeding value prediction and for assisting in selection decisions. In this context, more research efforts in breeding and genetics are focusing more on genome annotation and phenotyping. In animal breeding, the term phenomics is almost exchangeable with high-throughput phenotyping and automated phenotyping, i.e., the recording of high dimensional phenotypic data on an organism-wide scale. High throughput phenotyping is achieved through using a variety of sensors. The sensors generate raw data that must be captured, processed, cleaned, and modeled to produce meaningful phenotypes. Animal breeders are used to working with large amounts of data. However, high throughput phenomics usually produces data that may quickly overwhelm storage and processing capabilities of even the most prepared data analysis teams. Moreover, the black-box nature of many algorithms used to process raw sensor data into actionable phenotypes usually require thorough validation and checking. Animal breeders may build on prior lessons learned processing and modeling high-throughput genotypes and traditional phenotypes and we also need to adopt new practices commonly used in the field of machine learning and computer science. Finally, high throughput phenomics will produce novel phenotypes that will require different modeling or revisiting traditional models. In this talk I will share my views on all these points, and I will illustrate them using literature data and data from my own research related to computer vision, modeling of social genetic effects and modeling of social interactions.

Over the recent years smart cities have been emerged as promising paradigm for a transition toward providing effective and real time smart services. Despite the great potential it brings to citizens' life, security and privacy issues still need to be addressed. Due to technology advances, large amount of data is produced, where machine learning methods are applied to learn meaningful patterns. In this paper a machine learning-based framework is proposed for detecting distributed Denial of Service (DDoS) attacks in smart cities. The proposed framework applies restricted Boltzmann machines to learn high-level features from raw data and on top of these learned features, a feed forward neural network model is trained for attack detection. The performance of the proposed framework is verified using a smart city dataset collected from a smart water plant. The results show the effectiveness of the proposed framework in detecting DDoS attacks.

Software products are often released with missing functionality, errors, or incompatibilities that may result in failures in the field, inferior performances, or, more generally, user dissatisfaction. In previous work, we presented the GAMMA technology, which facilitates remote analysis and measurement of deployed software and allows for gathering programexecution data from the field. When monitoring a high number of deployed instances of a software product, however, a large amount of data is collected. Such raw data are useless in the absence of a suitable data-mining and visualization technique that supports exploration and understanding of the data. In this paper, we present a new technique for collecting, storing, and visualizing program-execution data gathered from deployed instances of a software product. We also present a prototype toolset, GAMMATELLA, that implements the technique. We show how the visualization capabilities of GAMMATELLA allows for effectively investigating several kinds of execution-related information in an interactive fashion.

Traffic congestion is a costly phenomenon of every-day life. Reinforcement Learning (RL) is a promising solution due to its applicability to solving complex decision-making problems in highly dynamic environments. To train smart traffic lights using RL, large amounts of data is required. Recent RL-based approaches consider training to occur on some nearby server or a remote cloud server. However, this requires that traffic lights all communicate their raw data to some central location. For large road systems, communication cost can be impractical, particularly if traffic lights collect heavy data (e.g., video, LIDAR). As such, this work pushes training to the traffic lights directly to reduce communication cost. However, completely independent learning can reduce the performance of trained models. As such, this work considers the recent advent of Federated Reinforcement Learning (FedRL) for edge-enabled traffic lights so they can learn from each other's experience by periodically aggregating locally-learned policy network parameters rather than share raw data, hence keeping communication costs low. To do this, we propose the SEAL framework which uses an intersection-agnostic representation to support FedRL across traffic lights controlling heterogeneous intersection types. We then evaluate our FedRL approach against Centralized and Decentralized RL strategies. We compare the reward-communication trade-offs of these strategies. Our results show that FedRL is able to reduce the communication costs associated with Centralized training by 36.24%; while only seeing a 2.11 % decrease in average reward (i.e., decreased traffic congestion).

Data visualization tools are important tools for making health decisions based on large amounts of data. However, very few comparisons of data visualization tools have been done in the context of health-related research. This paper compares commonly available visualization tools from the perspective of a dashboard developer in the context of scientific health research using real-world data and to provide general suggestions for dashboard developers’ future visualization efforts in health-related scientific discoveries. We evaluated four commonly available visualization tools (Tableau, Spotfire, PowerBI, and R Shiny). Each tool was evaluated on visualization outcomes (comparing the generated outputs of tables, scatterplots, correlation plots, treemaps, heatmaps, and geographic maps), usability, Cloud compatibility, analytic integration, and a few other factors. All tools generated comparable visualization outcomes in our tests and were equally usable with the exception of R Shiny which is not recommended for users with limited coding experience. Every tool except R Shiny supports connections to PostgreSQL, gSheets, and Athena without additional configurations. All tools support integration with R and Python; however, PowerBI requires some additional configuration. Every tool is capable of data pre-processing and supports data transformation, null detection and outlier detection. Each tool has its own unique advantages and limitations.

A hybrid compression method for the NMR data based on window averaging and Discrete Cosine Transform