SWeeP: representing large biological sequences datasets in compact vectors

Machine learning for Big Data analytics in plants.

Abstract. In European phytosociology, national classifications of corresponding vegetation types show considerable differences even between neighbouring countries. Therefore, the European Vegetation Survey project urgently needs numerical classification methods for large data sets that are able to produce compatible classifications using data sets from different countries. We tested the ability of two methods, TWINSPAN and COCKTAIL, to produce similar classifications of wet meadows (Calthion, incl. Filipendulenion) for Germany (7909 relevés) and the Czech Republic (1287 relevés) in this respect.In TWINSPAN, the indicator ordination option was used for classification of two national data sets, and the extracted assignment criteria (indicator species) were applied crosswise from one to the other national data set. Although the data sets presumably contained similar community types, TWINSPAN revealed almost no correspondence between the groups derived from the proper classification of the national data set and the groups defined by the assignment criteria taken from the other national data set. The reason is probably the difference in structure between the national data sets, which is a typical, but hardly avoidable, feature of any pair of phytosociological data sets. As a result, the first axis of the correspondence analysis, and consequently the first TWINSPAN division, are associated with different environmental gradients; the difference in the first division is transferred and multiplied further down the hierarchy.COCKTAIL is a method which produces relevé groups on the basis of statistically formed species groups. The user determines the starting points for the formation of species groups, and groups already found in one data set can be tested for existence in the other data set. The correspondence between the national classifications produced by COCKTAIL was fairly good. For some relevé groups, the lack of correspondence to groups in the other national data set could be explained by the absence of the corresponding vegetation types in one of the countries, rather than by methodological problems.

Astronomy datasets are generally terabytes in size and contain hundreds of millions of objects separated into millions of files-factors which makes many analyses impractical to perform on small computers. The key question we answer in this paper is: How can we leverage Grid resources to make the analysis of large astronomy datasets a reality for the astronomy community? To address this question, we have developed a Web Services-based system, AstroPortal, that uses grid computing to federate large computing and storage resources for dynamic analysis of large datasets. Building on the GT4, we have built a prototype and implemented a first analysis, stacking, that sums multiple regions of the sky, a function that can help both identify variable sources and detect faint objects. AstroPortal gives the astronomy community a new tool to advance their research and to open new doors to opportunities never before possible on such a large scale.

The Parsimony Ratchet1 is presented as a new method for analysis of large data sets. The method can be easily implemented with existing phylogenetic software by generating batch command files. Such an approach has been implemented in the programs DADA (Nixon, 1998) and Winclada (Nixon, 1999). The Parsimony Ratchet has also been implemented in the most recent versions of NONA (Goloboff, 1998). These implementations of the ratchet use the following steps: (1) Generate a starting tree (e.g., a "Wagner" tree followed by some level of branch swapping or not). (2) Randomly select a subset of characters, each of which is given additional weight (e.g., add 1 to the weight of each selected character). (3) Perform branch swapping (e.g., "branch-breaking" or TBR) on the current tree using the reweighted matrix, keeping only one (or few) tree. (4) Set all weights for the characters to the "original" weights (typically, equal weights). (5) Perform branch swapping (e.g., branch-breaking or TBR) on the current tree (from step 3) holding one (or few) tree. (6) Return to step 2. Steps 2-6 are considered to be one iteration, and typically, 50-200 or more iterations are performed. The number of characters to be sampled for reweighting in step 2 is determined by the user; I have found that between 5 and 25% of the characters provide good results in most cases. The performance of the ratchet for large data sets is outstanding, and the results of analyses of the 500 taxon seed plant rbcL data set (Chase et al., 1993) are presented here. A separate analysis of a three-gene data set for 567 taxa will be presented elsewhere (Soltis et al., in preparation) demonstrating the same extraordinary power. With the 500-taxon data set, shortest trees are typically found within 22 h (four runs of 200 iterations) on a 200-MHz Pentium Pro. These analyses indicate efficiency increases of 20×-80× over "traditional methods" such as varying taxon order randomly and holding few trees, followed by more complete analyses of the best trees found, and thousands of times faster than nonstrategic searches with PAUP. Because the ratchet samples many tree islands with fewer trees from each island, it provides much more accurate estimates of the "true" consensus than collecting many trees from few islands. With the ratchet, Goloboff's NONA, and existing computer hardware, data sets that were previously intractable or required months or years of analysis with PAUP* can now be adequately analyzed in a few hours or days.

Many papers have presented rendering techniques and simplification ideas with the objective of speeding up image generation for irregular grid data sets. For large data sets, however, even the current fastest algorithms are known to require seconds to generate each image, making real time analysis of such data sets very difficult, or even impossible, unless one has access to powerful and expensive computer hardware. In order to synthesize a system for handling very large data set analysis, we have assembled algorithms for rendering, simplification and triangulation, and added to them some optimizations. We have made some improvements on one of the best current algorithms for rendering irregular grids, and added to it some simple approximation methods in both image and object space, resulting in a system that achieves high frame rates, even on slow computers without any specific graphic hardware. The algorithm adapts itself to the time budget it has available for each image generation, using hierarchical representations of the mesh for faster delivery of images when transformations are imposed to the data. When given additional time, the algorithm generates finer images, obtaining the precise final image if given sufficient time. We were able to obtain frame rates of the order of 5 Hz for medium-sized data sets, which is about 20 times faster than previous rendering algorithms. With a trade-off between image accuracy and speed, similar frame rates can be achieved on different computers.

With the advent of next-generation sequencing technologies, large data sets of several thousand loci from multiple conspecific individuals are available. Such data sets should make it possible to obtain accurate estimates of population genetic parameters, even for complex models of population history. In the analyses of large data sets, it is difficult to consider finite-sites mutation models (FSMs). Here, we use extensive simulations to demonstrate that the inclusion of FSMs is necessary to avoid severe biases in the estimation of the population mutation rate θ, population divergence times, and migration rates. We present a new version of Jaatha, an efficient composite-likelihood method for estimating demographic parameters from population genetic data and evaluate the usefulness of Jaatha in two biological examples. For the first application, we infer the speciation process of two wild tomato species, Solanum chilense and Solanum peruvianum. In our second application example, we demonstrate that Jaatha is readily applicable to NGS data by analyzing genome-wide data from two southern European populations of Arabidopsis thaliana. Jaatha is now freely available as an R package from the Comprehensive R Archive Network (CRAN).

SummaryAnalytical single-well models have been particularly useful in forecasting production rates and estimated ultimate recovery (EUR) for the massive number of wells in unconventional reservoirs. In this work, a physics-based decline-curve model accounting for linear flow and material balance in horizontal multistage-hydraulically-fractured wells is introduced. The main characteristics of pressure diffusion in the porous media and the fact that the reservoir is a limited resource are embedded in the functional form, such that there is a transition from transient to boundary-dominated flow and the EUR is always finite. Analogously to the frequently used Arps (1945) hyperbolic model, the new model has only three parameters, where two of them define the decline profile and the third one is a multiplier.This model is applied to a large data set in a work flow that incorporates heuristic knowledge into the history matching and uncertainty quantification by assigning weights to rate measurements. The heuristic rules aim to lessen the effects of nonreservoir-related variations in the production data (e.g., temporary shut-in caused by fracturing in a neighboring well) and emphasize the reservoir dynamics to perform reliable predictions. However, there are additional degrees of freedom in the way these rules define the values of the weights; therefore, a criterion is established that “calibrates” the uncertainty in the probabilistic models by adjusting the parameters in the heuristic rules. Uncertainty quantification and calibration are performed using a Bayesian approach with hindcasts. This methodology is implemented in an automated framework and applied to 992 gas wells from the Barnett Shale. A comparison with the Arps (1945) hyperbolic model, the Duong (2011) model, and stretched exponential model for this data set shows that the new model is the most conservative in terms of estimated reserves.

BackgroundThe genetic and molecular basis for many intermediate and end stage phenotypes in model systems such as C. elegans and D. melanogaster has long been known to involve pleiotropic effects and complex multigenic interactions. Gene sets are groups of genes that contribute to multiple biological or molecular phenomena. They have been used in the analysis of large molecular datasets such as microarray data, Next Generation sequencing, and other genomic datasets to reveal pleiotropic and multigenic contributions to phenotypic outcomes. Many model systems lack species specific organized phenotype based gene sets to enable high throughput analysis of large molecular datasets.Results and discussionHere, we describe two novel collections of gene sets in C. elegans and D. melanogaster that are based exclusively on genetically determined phenotypes and use a controlled phenotypic ontology. We use these collections to build genome-wide models of thousands of defined phenotypes in both model species. In addition, we demonstrate the utility of these gene sets in systems analysis and in analysis of gene expression-based molecular datasets and show how they are useful in analysis of genomic datasets connecting multigenic gene inputs to complex phenotypes.ConclusionsPhenotypic based gene sets in both C. elegans and D. melanogaster are developed, characterized, and shown to be useful in the analysis of large scale species-specific genomic datasets. These phenotypic gene set collections will contribute to the understanding of complex phenotypic outcomes in these model systems.

Environmental monitoring programs provide large multivariate data sets that usually cover considerable spatial and temporal variabilities. The apparent complexity of these data sets requires sophisticated tools for their processing. Usually, fixed schemes are followed, including the application of numerical models, which are increasingly implemented in decision support systems. However, these schemes are too rigid with respect to detecting unexpected features, like the onset of subtle trends, non-linear relationships or patterns that are restricted to limited sub-samples of the total data set. In this study, an alternative approach is followed. It is based on an efficient non-linear visualization of the data. Visualization is the most powerful interface between computer and human brain. The idea is to apply an efficient and model-free tool, meaning without the necessity of prior assumptions about key properties of the data, such as dominant processes. In other words, processing of the data aimed at preserving a maximum amount of information and to leave it to the expert which features to analyze in more detail. A comprehensive data set from a 15-year monitoring program in the Lehstenbach watershed was used. The watershed is located in the Fichtelgebirge area, a mountainous region in South Germany, where land-use is forestry. Streamwater and groundwater have been monitored at 38 sampling sites, comprising 13 parameters. The data set was analyzed using a self-organizing map (SOM), combined with Sammon’s mapping. The 2D non-linear projection represented 89% of the variance of the data set. The visualization of the data set enabled an easy detection of outliers, assessing spatial versus temporal variance, and verifying a predefined classification of the sampling sites. Contamination of two of the observation wells was detected. Long-term trends of solute concentration in the catchment runoff could be differentiated from short-term dynamics, and a long-term shift in the dynamics was determined for different flow regimes individually. This analysis helped considerably to better understand the system’s behavior, to detect “hot spots” and to organize subsequent analyses of the data in a very efficient way.

Biological sequence usually contains yet to find knowledge, and mining biological sequences usually involves a huge dataset and long computation time. Common tasks for biological sequence mining are pattern discovery, classification and clustering. The newly developed model, Plausible Neural Network (PNN), provides an intuitive and unified architecture for such a large dataset analysis. This paper introduces the basic concepts of the PNN, and explains how it is applied to biological sequence mining. The specific task of biological sequence mining, exon/intron prediction, is implemented by using PNN. The experimental results show the capability of solving biological sequence mining tasks using PNN.

BackgroundTerminal restriction fragment length polymorphism (T-RFLP) analysis is a DNA-fingerprinting method that can be used for comparisons of the microbial community composition in a large number of samples. There is no consensus on how T-RFLP data should be treated and analyzed before comparisons between samples are made, and several different approaches have been proposed in the literature. The analysis of T-RFLP data can be cumbersome and time-consuming, and for large datasets manual data analysis is not feasible. The currently available tools for automated T-RFLP analysis, although valuable, offer little flexibility, and few, if any, options regarding what methods to use. To enable comparisons and combinations of different data treatment methods an analysis template and an extensive collection of macros for T-RFLP data analysis using Microsoft Excel were developed.ResultsThe Tools for T-RFLP data analysis template provides procedures for the analysis of large T-RFLP datasets including application of a noise baseline threshold and setting of the analysis range, normalization and alignment of replicate profiles, generation of consensus profiles, normalization and alignment of consensus profiles and final analysis of the samples including calculation of association coefficients and diversity index. The procedures are designed so that in all analysis steps, from the initial preparation of the data to the final comparison of the samples, there are various different options available. The parameters regarding analysis range, noise baseline, T-RF alignment and generation of consensus profiles are all given by the user and several different methods are available for normalization of the T-RF profiles. In each step, the user can also choose to base the calculations on either peak height data or peak area data.ConclusionsThe Tools for T-RFLP data analysis template enables an objective and flexible analysis of large T-RFLP datasets in a widely used spreadsheet application.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-014-0361-7) contains supplementary material, which is available to authorized users.

KINN: An alignment-free accurate phylogeny reconstruction method based on inner distance distributions of k-mer pairs in biological sequences

Advances in mass spectrometry-based proteomics have yielded a substantial mapping of the tyrosine phosphoproteome and thus provided an important step toward a systematic analysis of intracellular signaling networks in higher eukaryotes. In this study we decomposed an uncharacterized proteomics data set of 481 unique phosphotyrosine (Tyr(P)) peptides by sequence similarity to known ligands of the Src homology 2 (SH2) and the phosphotyrosine binding (PTB) domains. From 20 clusters we extracted 16 known and four new interaction motifs. Using quantitative mass spectrometry we pulled down Tyr(P)-specific binding partners for peptides corresponding to the extracted motifs. We confirmed numerous previously known interaction motifs and found 15 new interactions mediated by phosphosites not previously known to bind SH2 or PTB. Remarkably, a novel hydrophobic N-terminal motif ((L/V/I)(L/V/I)pY) was identified and validated as a binding motif for the SH2 domain-containing inositol phosphatase SHIP2. Our decomposition of the in vivo Tyr(P) proteome furthermore suggests that two-thirds of the Tyr(P) sites mediate interaction, whereas the remaining third govern processes such as enzyme activation and nucleic acid binding.

Dimensionality-reduction methods are a fundamental tool in the analysis of large datasets. These algorithms work on the assumption that the intrinsic of the data is generally much smaller than the ambient dimension in which it is collected. Alongside their usual purpose of mapping data into a smaller-dimensional space with minimal information loss, dimensionality-reduction techniques implicitly or explicitly provide information about the dimension of the dataset.In this paper, we propose a new statistic that we call the kappa-profile for analysis of large datasets. The kappa-profile arises from a dimensionality-reduction optimization problem: namely that of finding a projection that optimally preserves the secants between points in the dataset. From this optimal projection we extract kappa, the norm of the shortest projected secant from among the set of all normalized secants. This kappa can be computed for any dimension k; thus the tuple of kappa values (indexed by dimension) becomes a kappa-profile. Algorithms such as the Secant-Avoidance Projection algorithm and the Hierarchical Secant-Avoidance Projection algorithm provide a computationally feasible means of estimating the kappa-profile for large datasets, and thus a method of understanding and monitoring their behavior. As we demonstrate in this paper, the kappa-profile serves as a useful statistic in several representative settings: weather data, soundscape data, and dynamical systems data.

Background: Integrative conjugative elements (ICEs) play a significant role in the dissemination of antibiotic resistance genes among Vibrio cholerae strains. However, there are currently no standardized methods for ICE typing that allow for the analysis of large genomic datasets. Aim: To conduct a comparative analysis of ICE sequences in Vibrio cholerae strains of various origins and to develop an algorithm for their typing. Materials and methods: The study utilized whole-genome sequencing data from 120 toxigenic (ctxAB+tcpA+) V. cholerae O1 El Tor strains obtained using the MiSeq platform (Illumina, USA) and MinION platform (Oxford Nanopore, UK), as well as data from NCBI databases (1,886 genomes) and the European Nucleotide Archive (441 strains). The software for ICE detection and typing was developed in Java (version 11.0.13) and is available at: http://antiplague.ru/ice-genotyper/. Results: A comparative analysis of ICE elements in toxigenic V. cholerae strains was performed. An ICE typing algorithm based on gene composition was proposed. Analysis of the V. cholerae genome collection revealed three previously undescribed ICE elements, designated ICEVchRus1, ICEVchHai3, and ICEVchLaos. Conclusions: The study identified three previously undescribed ICE elements and mapped their distribution across Russia and other regions of the world. It was established that during the cholera outbreak in Dagestan in 1994, strains containing ICEVchBan11 and ICEVchBan9 were circulating simultaneously.