A data-aware cognitive engine for scheduling data intensive applications in a grid

In data-intensive applications, such as high-energy physics, bio-informatics, we encounter applications involving numerous jobs that access and generate large datasets. Effective scheduling of such applications is a challenge, due to the need to consider for both computational resources and data storage resources. In this paper, we describe an adaptive scheduling model that considers availability of computational, storage and network resources. Based on this model we implement a scheduler used in our campus grid. The results achieved by our scheduler have been analysed by comparing with greedy algorithm that is widely used in computational grids and some data grids.

QoS-aware replica placement for data intensive applications

One of the advantages of following a MDA-based approach in the development of interactive applications is the possibility of generating multiple platform-specific user interfaces (UI) from the same platform independent UI model. However, the effort required to create the UI model may be significant. In the case of data-intensive applications, a large part of the UI structure and functionality is closely related with the structure and functionality of the domain entities described in the domain model, and the access rules specified in the use case model. This paper presents an approach to reduce the effort required to create platform independent UI models for data intensive applications, by automatically generating an initial UI model from domain and use case models. For that purpose, UML-aligned metamodels for domain and use case models are defined, together with a MOF-based metamodel for user interface models. The transformation rules that drive the UI model generation are introduced. It is also proposed a MDA-based process for the development of data intensive interactive applications based on the proposed model architecture and transformations.

Enhanced Dynamic Hierarchical Replication and Weighted Scheduling Strategy in Data Grid

List based scheduling algorithms have been proven an optimistic strategy with a shorter response time to generate feasible solutions for the workflow scheduling problem. Data-intensive and computation-intensive workflow applications have different characteristics in terms of the ratio between data transfer time and task execution time. Workflow scheduling algorithms in a cloud-based environment should adequately consider the characteristics of the underlying cloud platform such as the on-demand resource provisioning strategy, the practically unlimited compute capacities, the booting times of virtual machines, the homogeneous network and the pay-as-you-go price model to produce an optimal scheduling solution within the deadline constraint of a given workflow. In this paper, a path based scheduling algorithm, named LPOD, is proposed to find the best workflow schedule solution with minimum monetary cost in a cloud computing environment. A series of case studies have been carefully conducted using synthetic workflows based on DATAVIEW, which is a popular open-source big data workflow management system. The experimental results show that the proposed algorithm is efficient and can generate better workflow schedules than the state-of-the-art algorithms such as IC-PCP and SGX-E2C2D.

Power, especially that consumed for storing data, and cooling costs for data centers have increased rapidly. The main applications running at data centers are data intensive applications such as large file servers or database systems. Recently, power management of the data intensive applications has been emphasized in the literature. Such reports discuss the importance of power savings. However, these reports lack research on power management models for the efficient use of data intensive applications' I/O behaviors. This paper proposes a novel energy efficient storage management system that monitors both application- and device-level I/O patterns at run time, and uses not only the device-level I/O pattern but also application level patterns. First, the design of the proposed model combined with such large data intensive applications will be shown. The key features of the model are i) classifying application-level I/O into four patterns using run-time access behaviors such as the length of idle time and read/write frequency, and ii) adopting an appropriate power-saving method-based on these application level I/O patterns. Next, the proposed method is quantitatively evaluated with typical data intensive applications such as file servers, OLTP, and DSS. It is shown that energy efficient storage management is effective in achieving large power savings compared with traditional approaches while an application is running.

Cognitive radio system (CRS) technology can help respond to the growing mobile traffic demand by improving network resource usage and gaining access to new shared spectrum bands. This article presents a cognitive radio trial environment (CORE) consisting of cognitive engines (CEs) to control different radio systems [e.g., long-term evolution (LTE) and wireless open-access research platform (WARP)-based networks]. Load balancing and authorized shared access (ASA) are demonstrated using the trial environment with promising results. The ASA-based spectrum sharing trial is presented for the first time with a real-life mobile network accessing a shared spectrum band under a licensed shared access (LSA) regime. Cognitive decision making brings intelligence to the usage of the radio and network resources and, at best, increases considerably end users? quality of service (QoS) compared to the standard systems, as shown for QoS-based offloading.

A grid consists of high-end computational, storage, and network resources that, while known a priori, are dynamic with respect to activity and availability. Efficient scheduling of requests to use grid resources must adapt to this dynamic environment while meeting administrative policies. This paper discusses the necessary requirements of such a scheduler and proposes a framework that can administrate grid policies and schedule complex and data intensive scientific applications. We present early experimental results for proposed a framework that effectively utilizes other grid infrastructure such as workflow management systems and execution systems. These results demonstrate that proposed a framework can effectively schedule work across a large number of distributed clusters that are owned by multiple units in a virtual organization.KeywordsDirected Acyclic GraphGrid ResourceGrid EnvironmentSchedule SystemVirtual OrganizationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Recent advances in unsteady flow visualization

Abstract: Processing and analyzing large volumes of data plays an increasingly important role in many domains of scientific research. We are developing a compiler that processes data intensive applications written in a dialect of Java and compiles them for efficient execution on distributed memory parallel machines. In this paper, we focus on the problem of generating correct and efficient communication for data intensive applications. We present static analysis techniques for 1) extracting a global reduction function from a data parallel loop, and 2) determining if a subscript function is monotonic. We also present a runtime technique for reducing the volume of communication during the global reduction phase. We have experimented with two data intensive applications to evaluate the efficacy of our techniques. Our results show that 1) our techniques for extracting global reduction functions and establishing monotonicity of subscript functions can successfully handle these applications, 2) significant reduction in communication volume and execution times is achieved through our runtime analysis technique, 3) runtime communication analysis is critical for achieving speedups on parallel configurations.

Abstract: Processing and analyzing large volumes of data plays an increasingly important role in many domains of scientific research. We are developing a compiler that processes data intensive applications written in a dialect of Java and compiles them for efficient execution on distributed memory parallel machines. In this paper, we focus on the problem of generating correct and efficient communication for data intensive applications. We present static analysis techniques for 1) extracting a global reduction function from a data parallel loop, and 2) determining if a subscript function is monotonic. We also present a runtime technique for reducing the volume of communication during the global reduction phase. We have experimented with two data intensive applications to evaluate the efficacy of our techniques. Our results show that 1) our techniques for extracting global reduction functions and establishing monotonicity of subscript functions can successfully handle these applications, 2) significant reduction in communication volume and execution times is achieved through our runtime analysis technique, 3) runtime communication analysis is critical for achieving speedups on parallel configurations.

Many geographic information systems (GIS) and computer mapping packages for graphic display provide data enhancement routines that utilize spatial filtering and may even include sample filter operators. With care, spatial filtering or convolving a filter with string or gridded data can be very effective in enhancing digital information. Filtering is intended to highlight specific features, remove unwanted noise or restore an image. The filtered output may be smoothed, it may portray features of a specific size, or it may subject data to mathematical operations. Results depend on the form of the filter operator, Fourier analysis of the filter operator produces frequency and phase diagrams that describe the effect the filter will have on the input data. These diagrams indicate how the component frequencies in the input data will be altered by the filter. Phase relates to the position of contained features, and amplitudes are responsible for feature definition. Derivative filtering may alter both phase and amplitude and completely change the appearance of the output. Spatial filtering can be very effective, however the interpreter must exercise great care selecting suitable filters. Fourier analysis can aid in design of a suitable filter and in the resultant evaluation. Examples herein describe some common three-value filters for string data and three-by-three matrix filters for grid and raster data. Digital spatial filters are arrays of values with the same form and interval as the original numeric data set (e.g., Robinson and Treitel, 1969). Filters may be strings of one-dimensional values, two-dimensional gridded data sets, or even multi-dimensional data. Spatial filters to be used with digital data sets can be designed and evaluated in either the spatial or the frequency domain. Similarly, line, map, and image data can be transformed and viewed in both domains. Fourier analysis is the bridge between the two domains (Cooley and Tukey, 1965; Gentleman and Sande, 1966; Robinson and Cohn, 1979; Sondegard, Robinson and Merriam, 1989). The filtering operation can be carried out in either domain by convolution in the spatial domain or by multiplication in the frequency domain. Convolution is the Fourier transform of multiplication.

This paper examines the performance of three gridded precipitation data sets, namely, Global Precipitation Climatology Centre (GPCC), Tropical Precipitation Measuring Mission (TRMM), and the Modern-Era Retrospective Analysis for Research and Applications (MERRA), for the duration of 25years using 9 rain gauge data sets of the Sina basin, India. Statistical measures were employed to measure the performance in reproducing the rainfall and to assess its ability to detect the rainfall/no rainfall events, its structure, pattern, and spatio-temporal variations in the monthly and annual time scales. Compromise programming (CP) is used to rank the statistical performances of selected gridded precipitation data sets and found that TRMM attained first rank for the 8 stations followed by MERRA. The precipitation concentration index (PCI) checks the pattern and distribution of rainfall and found that observed data shows a uniform distribution in the basin; however, all the three gridded data sets failed to demonstrate uniform distribution. Categorical metrics like Probability of Detection (POD) and False Alarm Ratio (FAR) revealed that TRMM followed by MERRA and GPCC have good capabilities to detect rainfall/no rainfall events at different thresholds. All the trends drawn between observed data set and gridded precipitation data sets revealed that the MERRA data tend to underestimate and the TRMM and GPCC data tend to overestimate the values and intensities of rainfall data sets at most of the stations for both monthly and annual time scales. The data analysis of extreme rainfall points at monthly and annual time scales exhibits better performance of TRMM data sets. Overall, the TRMM data set is capable in replicating different characteristics of the observed data in the study area and could be used for hydro-meteorological and climatic studies when continuous observed data set is not available.

The application of wearable devices is promoting the development toward digitization and intelligence in the field of health. However, the current smart devices centered on human health have disadvantages such as weak perception, high interference degree, and unfriendly interaction. Here, an intelligent health agent based on multifunctional fibers, with the characteristics of autonomy, activeness, intelligence, and perceptibility enabling health services, is proposed. According to the requirements for healthcare in the medical field and daily life, four major aspects driven by intelligent agents, including health monitoring, therapy, protection, and minimally invasive surgery, are summarized from the perspectives of materials science, medicine, and computer science. The function of intelligent health agents is realized through multifunctional fibers as sensing units and artificial intelligence technology as a cognitive engine. The structure, characteristics, and performance of fibers and analysis systems and algorithms are reviewed, while discussing future challenges and opportunities in healthcare and medicine. Finally, based on the above four aspects, future scenarios related to health protection of a person's life are presented. Intelligent health agents will have the potential to accelerate the realization of precision medicine and active health.

The data grid is one of the infrastructures for data and storage resource management. Data replication can improve data access performance, fault tolerance, and reduce network transmission bandwidth. As the site storage space is limited in the grid, how to effectively use grid resources become an important challenge. This paper introduces a dynamic grid replication algorithm based on popularity support and confidence (BPSC). Through the algorithm, data and its associated copy can be placed on a suitable site, with reduced access latency. Through Optorsim, simulation results show that the algorithm can provide better performance compared with other algorithms Grid computing is an important branch of distributed computing. Grid computing constructs a virtual computer system through using a different location of the large number of heterogeneous resources (CPU and storage), provides the model to solve the problem of large scale computing and storage. The data grid is mainly the grid for data storage management. The data grid is defined as an effective combination of data and computing resources of distributed systems by Data Grid Group. The large-scale data- intensive applications, such as bioinformatics, high energy physics, space information generated vast amounts of data, these data may be TB scale, even up to the scale of PB. It is not practical to maintain large scale data at each site, and provide computing resources for visitors. At the same time, it will lead to increased access latency through a centralized way to storing and accessing these data from each site. Data replication technology is an effective method to reduce access latency in the data grid system. Data replication is one of the basic methods of distributed systems to improve the efficiency of system access, maintenance, system high availability. Data replication is to store data in the appropriate site for users to effective access to massive data. Data replication can reduce data access latency, reduce network bandwidth consumption, avoid congestion, adjust the load balancing in server-side, improve system reliability. Many data replication algorithm can reduce data access latency, improve system reliability, but which data will be copied, when the data will be copied at, where the data will be copied are the basic means of determining data replication algorithm. In this paper, we introduced a real-time dynamic data replication algorithm. The algorithm is based on the popularity of file dynamic to replicate data, and consider the storage space limitations of the site. Through optorsim simulation verified the algorithm. This paper is organized as follows: Section 2 is about related works on replication in data grids. In Section 3, system model and component in system are presented. The file