Web 上の大規模データの解析による Moon & Spencer の色彩調和論における「美度」の妥当性の検証

A variety of real-world data and networks can be described by a heavy-tailed probability distribution of its values, vertex degrees, or other significant properties, that follows the power law. Such a scale-free data and networks can be found in both natural phenomena such as protein interaction networks and gene regulation networks and man-made structures like the Internet, language, and various social networks. An efficient analysis of large scale data and networks is often impractical and various heuristic and metaheuristc sampling techniques are deployed to select smaller subsets of the data for analysis and visualisation. A key goal of data and network sampling is to select such a subset of the original data that would accurately represent the original data with respect to selected attributes. In this work we propose a novel genetic algorithm for scale-free data and network sampling and evaluate the algorithm in a series of computational experiments.

Graph analytics has emerged as an important tool in the analysis of large scale data from diverse application domains such as social networks, cyber security and bioinformatics. Counting the number of triangles in a graph is a fundamental kernel with several applications such as detecting the community structure of a graph or in identifying important vertices in a graph. The ubiquity of massive datasets is driving the need to scale graph analytics on parallel systems. However, numerous challenges exist in efficiently parallelizing graph algorithms, especially on distributed-memory systems. Irregular memory accesses and communication patterns, low computation to communication ratios, and the need for frequent synchronization are some of the leading challenges. In this paper, we present TriC, our distributed-memory implementation of triangle counting in graphs using the Message Passing Interface (MPI), as a submission to the 2020 Graph Challenge competition. Using a set of synthetic and real-world inputs from the challenge, we demonstrate a speedup of up to 90 x relative to previous work on 32 processor-cores of a NERSC Cori node. We also provide details from distributed runs with up to 8192 processes along with strong scaling results. The observations presented in this work provide an understanding of the system-level bottlenecks at scale that specifically impact sparse-irregular workloads and will therefore benefit other efforts to parallelize graph algorithms.

The analysis of large scale data logged from complex cyber-physical systems, such as microgrids, often entails the discovery of invariants capturing functional as well as operational relationships underlying such large systems. We describe a latent factor approach to infer invariants underlying system variables and how we can leverage these relationships to monitor a cyber-physical system. In particular we illustrate how this approach helps rapidly identify outliers during system operation.

Recently, the importance of interest in color harmony is increasing. For this, the paper focuses on harmony between colors. Selecting colors by mere feelings, however, can bring about results that do not fit in with the overall concept and can cause differences according to social environment, conscience, gender, etc. However, learning the color harmony theory requires a lot of time since users have to know the color harmony theory, and train for color combination. To solve this problem, the paper present the method finding harmonious combinations based on color harmony theories using relational algebra. The method proposes the color harmony rule based on the color harmony theory and formalized the rule. Through this, users can produce the same results in color selection as professionals without the expert knowledge in color.KeywordsColorColor HarmonyColor Harmony TheoryRelational Algebra

The detection of useful patterns in large datasets has attracted considerable interest recently. The hierarchical K-means clustering algorithm (HKM) is very efficient in large scale data analysis. It has been extensively used for building visual vocabulary in large scale image/video retrieval. However, the accuracy and speed of HKM still have room for improvement. In this paper, we propose a Parallel N-path labeling HKM clustering algorithm (PNHKM) which improves on the HKM clustering algorithm in the following ways. Firstly, we adopt a Greedy N-best Paths Labeling (GNPL) method to improve the clustering accuracy. Secondly, we focus on developing a parallel clustering algorithm for multicore processors. Our results confirm that the PNHKM is much faster and more effective.

BackgroundIn many parts of the world, salt marshes play a key ecological role as the interface between the marine and the terrestrial environments. Salt marshes are also exceedingly important for public health as larval habitat for mosquitoes that are vectors of disease and significant biting pests. Although grid ditching and pesticides have been effective in salt marsh mosquito control, marsh degradation and other environmental considerations compel a different approach. Targeted habitat modification and biological control methods known as Open Marsh Water Management (OMWM) had been proposed as a viable alternative to marsh-wide physical alterations and chemical control. However, traditional larval sampling techniques may not adequately assess the impacts of marsh management on mosquito larvae. To assess the effectiveness of integrated OMWM and marsh restoration techniques for mosquito control, we analyzed the results of a 5-year OMWM/marsh restoration project to determine changes in mosquito larval production using GIS and geostatistical methods.MethodsThe following parameters were evaluated using "Before-After-Control-Impact" (BACI) design: frequency and geographic extent of larval production, intensity of larval production, changes in larval habitat, and number of larvicide applications. The analyses were performed using Moran's I, Getis-Ord, and Spatial Scan statistics on aggregated before and after data as well as data collected over time. This allowed comparison of control and treatment areas to identify changes attributable to the OMWM/marsh restoration modifications.ResultsThe frequency of finding mosquito larvae in the treatment areas was reduced by 70% resulting in a loss of spatial larval clusters compared to those found in the control areas. This effect was observed directly following OMWM treatment and remained significant throughout the study period. The greatly reduced frequency of finding larvae in the treatment areas led to a significant decrease (~44%) in the number of times when the larviciding threshold was reached. This reduction, in turn, resulted in a significant decrease (~74%) in the number of larvicide applications in the treatment areas post-project. The remaining larval habitat in the treatment areas had a different geographic distribution and was largely confined to the restored marsh surface (i.e. filled-in mosquito ditches); however only ~21% of the restored marsh surface supported mosquito production.ConclusionThe geostatistical analysis showed that OMWM demonstrated considerable potential for effective mosquito control and compatibility with other natural resource management goals such as restoration, wildlife habitat enhancement, and invasive species abatement. GPS and GIS tools are invaluable for large scale project design, data collection, and data analysis, with geostatistical methods serving as an alternative or a supplement to the conventional inference statistics in evaluating the project outcome.

BackgroundRecent advances in experimental and computational technologies have fueled the development of many sophisticated bioinformatics programs. The correctness of such programs is crucial as incorrectly computed results may lead to wrong biological conclusion or misguide downstream experimentation. Common software testing procedures involve executing the target program with a set of test inputs and then verifying the correctness of the test outputs. However, due to the complexity of many bioinformatics programs, it is often difficult to verify the correctness of the test outputs. Therefore our ability to perform systematic software testing is greatly hindered.ResultsWe propose to use a novel software testing technique, metamorphic testing (MT), to test a range of bioinformatics programs. Instead of requiring a mechanism to verify whether an individual test output is correct, the MT technique verifies whether a pair of test outputs conform to a set of domain specific properties, called metamorphic relations (MRs), thus greatly increases the number and variety of test cases that can be applied. To demonstrate how MT is used in practice, we applied MT to test two open-source bioinformatics programs, namely GNLab and SeqMap. In particular we show that MT is simple to implement, and is effective in detecting faults in a real-life program and some artificially fault-seeded programs. Further, we discuss how MT can be applied to test programs from various domains of bioinformatics.ConclusionThis paper describes the application of a simple, effective and automated technique to systematically test a range of bioinformatics programs. We show how MT can be implemented in practice through two real-life case studies. Since many bioinformatics programs, particularly those for large scale simulation and data analysis, are hard to test systematically, their developers may benefit from using MT as part of the testing strategy. Therefore our work represents a significant step towards software reliability in bioinformatics.

Automated video and image analysis can be a very efficient tool to behavior study, especially in hard access environments for researchers. The understanding of this social behavior can play a key role in the sustainable design of control policies of many species. This paper proposes the use of computer vision algorithms to identify and track, the Norway lobster, Nephrops norvegicus, a burrowing decapod with relevant commercial value which is captured by trawling. These animals can only be captured when are engaged in seabed excursions, which are strongly related with their social behavior. This emergent behavior is modulated by the day-night cycle, but social interactions remain unknown to the scientific community. The paper introduces an identification scheme made of four distinguishable black and white tags (geometric shapes). The project has recorded 15-day experiments in laboratory, under monochromatic blue light (472 nm.) and darkness conditions (recorded using Infra Red light). Using this massive image set, we propose a comparative of state-of-the-art computer vision algorithms to distinguish and track the different animals' movements. We evaluate the robustness to the high noise presence in the infrared video signals and free out-of-plane rotations due to animal movement. The experiments show promising accuracies under a cross-validation protocol, being adaptable to the automation and analysis of large scale data. In a second contribution, we created an extensive dataset of shapes (46027 different shapes) from four daily experimental video recordings, which will be available to the community.

This study proposes content and interaction analysis techniques for a large repository created from social media content. Though we have presented our study for a large platform dedicated to discussions around financial topics, the proposed methods are generic and applicable to all platforms. Along with an extension of topic extraction method using Latent Dirichlet Allocation, we propose a few measures to assess user participation, influence and topic affinities specifically. Our study also maps user-generated content to components of behavioral finance. While these types of information are usually gathered through surveys, it is obvious that large scale data analysis from social media can reveal many potentially unknown or rare insights. Characterising users based on their platform behavior to provide critical insights about how communities are formed and trust is established in these platforms using graphical analysis is also studied.

Accurate prediction of cycle time (CT) plays an important role in the promises of a good delivery-time for semiconductor manufacturers. However, the CTs of wafer lots are difficult to forecast since they are influent by a lot factors (e.g., workload for every machine). The identification of key factors (called CT-related) can not only improve the prediction performance but also facilitate the control of CT, which is of particular significance. This paper focuses on gathering subtle variables into the candidate variables set along with the further analysis which is required in the correlation analysis between various candidate factors and CTs of wafer lots. The candidates set is first constructed to collect all factors that may affect wafer lots’ CT. Then, a regression-based model is proposed to select CT-related variables, which consists of three parts: 1) discretization; 2) adaptive logistic regression-based correlation analysis; and 3) CT-related factor selection. Subsequently, a parallel computation model is implemented to forecast the CTs of wafer lots. In the numerical experiments, 108 CT-related factors stood out from 774 candidates and replaced six global factors (used as reference) to predict CTs of wafer lots. The results indicated that the proposed approach had higher accuracy than linear regression and back propagation network in CT forecasting in large scale data analysis.

The purpose of the workis to explore the current problems and prospects of mining solution, big web data in real time, as well as the possibility of practical implementation of Web Mining technology for big web data on a practical example.Materials and methods.The study included a review of bibliographic sources on big data mining. We used Web Mining technology for associative analysis of large web data, as well as computer modeling of the practical task of transaction analysis using a general-purpose scripting language (PHP).Results.During the work, the specifics of the Data Mining technology were described, and a modern approach to the analysis of large web data –Web Mining was analyzed. A brief classification of tasks solved using Web Mining technology is given. The problem of data mining of large web data in a general-purpose scripting language (PHP) has been solved: the lack of libraries for data mining, the difficult normalization of data to the form necessary for associative analysis, interaction with the database management system. Also, an example showing an approach to the mining of large web data was implemented. Based on the understanding of Web Mining technology and the described difficulties of analyzing web data in the PHP language, methods for effectively solving the practical problem of analyzing web data based on transactions committed in a dynamic web application have been proposed. A module for associative analysis of customer transactions in the programming language PHP was developed. The module includes an intelligent data processing class. The structural scheme of the module and system architecture were developed. The constructed module allows us to solve the main part of the problem of associative analysis of large web data using Web Mining technology in order to solve the problem of identifying patterns in a large array of web data. Associative analysis of web data is much faster because of the combination of a general-purpose scripting language and an object-oriented approach.Conclusion.According to the results of the study, it can be argued that the current state of the technology for the analysis of large web data allows efficiently process data objects, identify patterns, obtain hidden data and receive complete statistical data in real time. The results can be used both for the purpose of the initial research of technologies for analyzing large web data, and as an addition to the content management system for the intelligent analysis of web data. The usage of the technology of associative analysis and the created universal handler class makes the created module flexible, while the possibility of manual integration makes this module universal. With manual integration, the database management system is not important. Algorithm methods work with selected data. This factor greatly simplifies the further development of program code.

This study proposed a color scheme that is harmonious with the working environment of industrial sites using Birren's color harmony theory for color planning. To apply the Birren's color harmony theory to working clothes, the basic colors were chosen, and six of the eight harmony formulas of Birren excluding achromatic colors (white + grey + black) and solid color harmony (solid colors + white + black) were used to form a palette for each case. For the basic colors, the color chips of four dominant colors (yellow-green, sky-blue, blue, and violet), which were chosen from a field survey for preferences in the first step, and the production of materials in the second step were collected through the PANTONE color chips. The selected color chips were PANTONE 13-0550 TPX, PANTONE 15-4105 TPX, PANTONE 18-3949 TPX, and PANTONE 19-3720 TPX. These color chips were scanned and their RGB values were extracted through Photoshop CS. Then the colors were arranged in accordance with the Birren's color harmony formulas (Color+Tint+White, Color+Shade+Black, Tint+Tone+Shade, Shade+Tone+Black, Shade+Tone+White, and Tint+Shade+Tone+Gray). In addition, the proposed palette color schemes were applied through Birren's color harmony formulas using Texpro V 10.1 textile to the schematization of working clothes that were designed in the previous study. Palette formation in line with Birren's color harmony formulas provided scientific color arrangement results. Visually presenting the color scheme of working clothes will help the color selection of working clothes in tune with the circumstances of industrial sites.

After analysis and publication, there is no need to keep experimental data online on spinning disks. For reliability and costs inactive data is moved to tape and put into a data archive. The data archive must provide reliable access for at least ten years following a recommendation of the German Science Foundation (DFG), but many scientific communities wish to keep data available much longer. Data archival is on the one hand purely a bit preservation activity in order to ensure the bits read are the same as those written years before. On the other hand enough information must be archived to be able to use and interpret the content of the data. The latter is depending on many also community specific factors and remains an areas of much debate among archival specialists. The paper describes the current practice of archival and bit preservation in use for different science communities at KIT for which a combination of organizational services and technical tools are required. The special monitoring to detect tape related errors, the software infrastructure in use as well as the service certification are discussed. Plans and developments at KIT also in the context of the Large Scale Data Management and Analysis (LSDMA) project are presented. The technical advantages of the T10 SCSI Stream Commands (SSC-4) and the Linear Tape File System (LTFS) will have a profound impact on future long term archival of large data sets.

Big data analysis technology can be used to transform massive data of oil production engineering into knowledge that can be used to guide oilfield production. Due to the lack of suitable large data analysis tools, a large data analysis software system was developed to simplify the workload of data analysis. This paper illustrates the research and application of large energy consumption data analysis through the development of big data analysis methods, process improvement and software platform, then achieves the goal of reducing costs and improving efficiency.

<p>Over the last few years, the amount of large and complex data in the public domain has increased enormously and new challenges arose in the representation, analysis and visualization of such data. Considering the number of space missions that provided and will provide remote sensing data, there is still the need of a system that can be dispatched in several remote repositories and being accessible from a single client of commodity hardware.</p><p>To tackle this challenge, at the DLR Institute for Software Technology we have defined a dual backend frontend system, enabling the interactive analysis and visualization of large-scale remote sensing data. The basis for all visualization and interaction approaches is CosmoScout VR, a visualization tool developed internally at DLR, and publicly available on Github, that allows the visualization of complex planetary data and large simulation data in real-time. The dual component of this system is based on an MPI framework, called Viracocha, that enables the analysis of large data remotely, and allows the efficient network usage about sending compact and partial results for interactive visualization in CosmoScout as soon as they are computed.</p><p>A node-based interface is defined within the visualization tool, and this lets a domain expert to easily define customized pipelines for processing and visualizing the remote data. Each “node” of this interface is either linked with a feature extraction module, defined in Viracocha, or to a rendering module defined directly in CosmoScout. Being this interface completely customizable by a user, multiple pipelines can be defined over the same dataset to enhance even more the visualization feedback for analysis purposes.</p><p>Being an ongoing project, on top of these tools, as a novel strategy in EO data processing and visualization, we plan to define and implement strategies based on Topological Data Analysis (TDA). TDA is an emerging set of technique for processing the data considering its topological features. These include both the geometric information associated to a point, as well all the non-geometric scalar values, like temperature and pressure, to name a few, that can be captured during a monitoring mission. One of the major theories behind TDA is Discrete Morse Theory, that, given a scalar value, is used to define a gradient on such function, extract the critical points, identify the region-of-influence of each critical point, and so on. This strategy is parameter free and enables a domain scientist to process large datasets without a prior knowledge of it.</p><p>An interesting research question, that it will be investigated during this project is the correlation of changes of critical points at different time steps, and the identification of deformation (or changes) across time in the original dataset.</p>