Design of Data Management for Multi SPMD Workflow Programming Model

Software for scientific research purposes has received increased attention in recent years. Case studies have noted development practices, limitations, and problems in the development of scientific software. However, applicability of the results of these studies to improving the wider scientific software development practices is not known. This paper presents a survey of 60 scientific software developers. The survey was conducted online from August--September 2009, and aims to identify where improvements to scientific software practices can be made. While our results generally confirm previous work, we have found some notable differences. The use of IDEs and version control tools among the surveyed scientific software developers has increased, and trace-ability of scientific software is not as important to scientific software developers as it is to scientific software users. Documentation also appears to be more widely produced than previous studies indicate. However, there remains room for improvement in tool use, documentation, testing, and verification activities for scientific software development.

<p>Technical Debt (TD) is a software engineering metaphor that resembles the production of poor-<br> quality code to going into debt. In particular, a development team that ‘‘saves’’ effort while developing by<br> not removing inefficiencies, has to ‘‘pay-back’’ with interest, in the form of additional maintenance costs<br> (i.e., fixing bugs, adding features, etc.) due to the poor maintainability of the developed code. Although<br> maintainability assurance is an established practice in traditional software development (lately known as TD<br> management), it has still not attracted the attention of scientific software developers; i.e., researchers writing<br> code and developing tools for purely research purposes. Nevertheless, based on the literature and practice,<br> maintainability seems to be ranked as an important key-driver for the development of such applications; since<br> the effort needed to update the code before the experimentation (e.g., executing a simulation) is common<br> and should not receive low priority. In this paper, we present the outcome of a 3-year research project on<br> Technical Debt Management (TDM) for scientific software development. The outcome of the project is a<br> framework (termed: EXA2PRO TDM framework) and an accompanying platform for assisting scientific<br> software developers in managing the TD of their applications. The framework is a collection of methods<br> tailored for the mainstream programming languages of scientific software development, which have been<br> empirically validated through five pilot applications. The majority of the EXA2PRO framework suggestions<br> have been applied by scientific software developers and eased future maintenance activities.</p>

Introduction:Portable handheld computers and electronic data management systems have been used for national surveys in many high-income countries, however their use in developing countries has been challenging due to varying geographical, economic, climatic, political and cultural environments. In order to monitor and measure global adult tobacco use, the World Health Organization and the US Centers for Disease Control and Prevention initiated the Global Adult Tobacco Survey, a nationally representative household survey of adults, 15 years of age or older, using a standard core questionnaire, sample design, and data collection and management procedures. The Survey has been conducted in 14 low- and middle-income countries, using an electronic data collection and management system. This paper describes implementation of the electronic data collection system and associated findings.Methods:The Survey was based on a comprehensive data management protocol, to enable standardized, globally comparable high quality data collection and management. It included adaptation to specific country needs, selection of appropriate handheld hardware devices, use of open source software, and building country capacity and provide technical support.Results:In its first phase, the Global Adult Tobacco Survey was successfully conducted between 2008 and 2010, using an electronic data collection and management system for interviews in 302,800 households in 14 countries. More than 2,644 handheld computers were fielded and over 2,634 fieldworkers, supervisors and monitors were trained to use them. Questionnaires were developed and programmed in 38 languages and scripts. The global hardware failure rate was < 1% and data loss was almost 0%.Conclusion:Electronic data collection and management systems can be used effectively for conducting nationally representative surveys, particularly in low- and middle-income countries, irrespective of geographical, climatic, political and cultural environments, and capacity-building at the country level is an important vehicle for Health System Strengthening.

BackgroundA research area that has greatly benefited from the development of new and improved analysis technologies is Proteomics and large amounts of data have been generated by proteomic analysis as a consequence. Previously, the storage, management and analysis of these data have been done manually. This is, however, incompatible with the volume of data generated by modern proteomic analysis. Several attempts have been made to automate the tasks of data analysis and management. In this work we propose PRODIS (Proteomics Database Integrated System), a system for proteomic experimental data management. The proposed system enables an efficient management of the proteomic experimentation workflow, simplifies controlling experiments and associated data and establishes links between similar experiments through the experiment tracking function.ResultsPRODIS is fully web based which simplifies data upload and gives the system the flexibility necessary for use in complex projects. Data from Liquid Chromatography, 2D-PAGE and Mass Spectrometry experiments can be stored in the system. Moreover, it is simple to use, researchers can insert experimental data directly as experiments are performed, without the need to configure the system or change their experiment routine. PRODIS has a number of important features, including a password protected system in which each screen for data upload and retrieval is validated; users have different levels of clearance, which allow the execution of tasks according to the user clearance level. The system allows the upload, parsing of files, storage and display of experiment results and images in the main formats used in proteomics laboratories: for chromatographies the chromatograms and lists of peaks resulting from separation are stored; For 2D-PAGE images of gels and the files resulting from the analysis are stored, containing information on positions of spots as well as its values of intensity, volume, etc; For Mass Spectrometry, PRODIS presents a function for completion of the mapping plate that allows the user to correlate the positions in plates to the samples separated by 2D-PAGE. Furthermore PRODIS allows the tracking of experiments from the first stage until the final step of identification, enabling an efficient management of the complete experimental process.ConclusionsThe construction of data management systems for Proteomics data importing and storing is a relevant subject. PRODIS is a system complementary to other proteomics tools that combines a powerful storage engine (the relational database) and a friendly access interface, aiming to assist Proteomics research directly at data handling and storage.

Abstract. The increasing demand for software has generated greater need for software de-velopment processes. The literature, on the other hand, does not present processes for the development of scientific software, and, in this way, in previous studies, a Specific Process for Scientific Software (SPSS) was proposed. It was conceived based on the Humphrey’s Meth-odology, which defines that for the acquisition of a process it is necessary the execution of eight steps. Having in mind that the six first steps have already been concluded, the current work considers the execution of the seventh and eighth steps of the Methodology, which are the validation of the initial process and its later improvement. It is important to highlight the significance of these steps because, for a process to ensure its efficiency, it must be tested and validated. So, firstly, a webapp to automate and model the SPSS was developed, in order to help in its implementation. In the continuation of the study, SPSS was applied in real envi-ronments of scientific software development, to obtain results which allow them to be ana-lyzed through a validation technique, by inspection, named VProcInsp. At the end, in spite of some inconsistencies, it was concluded that the SPSS had indicated that it can really contrib-ute in the development of academic scientific software. The results are shown and discussed in this the article.

Genericity of parallel programming environments, enabling development of portable parallel programs, is expected to result in performance penalties. Furthermore, programmability and tool support of programming environments are important issues if a choice between programming environments has to be made. In this paper we propose a methodology to compare native and generic parallel programming environments, taking into account such competing issues as portability and performance. As a case study, this paper compares the Iserver-Occam, Parix, Express and PVM parallel programming environments on a 512-node Parasytec GCel. Furthermore, we apply our methodology to compare Parix and PVM on a new architecture, a 32-node Parsytec PowerXplorer, which is based on the PowerPC chip. In our approach we start with a representative application and isolate the basic (environment)-dependent building blocks. These basic building blocks, which depend on floating-point performance and communication capabilities of the environments, are analysed independently. We have measured point-to-point communication times, global communication times and floating-point performance. All information is combined into a time complexity analysis, allowing comparison of the environments on different degrees of functionality. Together with demands for portability of the code and development time (i.e. programmability), an overall judgement of the environments is given.

Parsec is a parallel programming environment whose goal is to simplify the development of multicomputer programs without, as is often the case, sacrificing performance. We have reconciled these objectives by “compiling” the structure of parallel applications into information to configure each of a small set of communication primitives on a context sensitive basis. In this chapter we show how Parsec can be used to implement a high-performance processor farm and compare Parsec and hand-optimized implementations to demonstrate that Parsec can achieve a similar level of performance. Extensive static analysis and optimization is necessary to achieve these results. We discuss both the tools which perform these tasks as well as the user interface that provides the necessary declarative structural information. Using the processor farm, we show how Parsec simplifies the task of specifying the structure of a parallel application and improves the result by supporting abstraction, reuse and scalability.

A computer adaptive language for the development of structural analysis programs

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

The healthcare industry has been using data and technology driven innovation for a long time now with Evidence Based Medicine (EBM) being at the helm. Effectively using technology is vital towards successful data management in healthcare. A convenient approach that can collect, store, evaluate, and analyze health data will be beneficial to all the stakeholders in the healthcare delivery chain. With the advancement, clubbed with the availability and affordability of technology, it has become an effective tool for providing the right- care backed with accurate, consolidated evidence. Such approach not only helps in improving the quality of care, it also helps in early detection of diseases and effective treatment, Big Data is providing the necessary data management tools for the same. A simple Extract-Transform-Load (ETL) procedure using Hadoop can be used. As Big data analyses large amounts of data to uncover hidden patterns, correlations and other insights, its role in making the right, predictive decisions for quality care of the patients and better health planning is possible. Big data technologies such as Hadoop bring significant cost advantages when it comes to storing large amounts of data and combined with the speed of Hadoop and in-memory analytics, with the ability to analyze new sources of data, we are able to analyze information instantly – and make decisions which is very essential in the healthcare provider segment. Hadoop does not only store, but also process any and many file data: big or small, can be plain text files or binary files like images and even multiple different version of some particular data format across different time periods. It consists of three core components: a distributed file system, a parallel programming framework, and a resource/job management system. In healthcare, patient records, health plans, insurance information and other types of information is extremely difficult to manage – but, this very data is full of key insights, the need is for an effective data management and analytics that can be pragmatic. By analyzing large amounts of information – both structured and unstructured – rapidly, health care providers can provide lifesaving diagnoses. Keywords: Big Data, EBM, ETL, Hadoop

Requirements engineering is crucial to the success of software development. However, in many scientific computing projects, traditional requirements engineering practices are ignored. We claim that there is a need for methodologies, which help capturing and managing requirements for these projects, to collaboratively develop scientific software with greater interoperability. We propose a model-based approach to elicit and manage requirements in scientific computing projects. The proposed approach is based on a meta-model in order to deal with the high complexity and frequent change in scientific software development. The meta-model also provides abstractions and notations targeted at scientific computing projects. The approach supports requirements engineering in these projects with the flexibility of easily managing requirements versioning, trace ability and communication across the boundary of disciplines. A tool prototype is implemented. To evaluate the proposed approach, we applied it in scientific computing projects. The approach has showed its advantages in the application and promotes software engineering in these projects.

Numerical Time-Dependent Partial Differential Equations for Scientists and Engineers

For decades, Lawrence Livermore National Laboratory (LLNL) has been engaged in significant research, development, and support for software to enable scientific computing and, particularly, the use of high performance computing (HPC) in the NNSA mission space. In particular, the move in the mid-1990’s to simulation as a leading component of stockpile stewardship through the ASCI and the successor ASC programs, as well as the need for reliable data acquisition and control software for the National Ignition Facility, have been important drivers in building expertise in production-quality software development at LLNL. LLNL has also been a leader in the DOE SciDAC FASTMath Institute and the DOE Exascale Computing Project (ECP), both of which have striven to make scientific computing software – in particular, the enabling technologies underpinning simulation capabilities – more widely adopted and sustainable. As such, we believe that our experience can inform the broader goal of software stewardship for scientific and high-performance computing. LLNL strongly supports the formation of a new DOE ASCR program element in software stewardship and sustainment. Historically, DOE ASCR has funded applied mathematics and computer science research that has led to the development of important new capabilities and algorithms that are expressed as artifacts in research software. Such frameworks, libraries, and tools have seldom been directly funded to address the important issues of code maintenance, documentation, robustness, and community building. Software engineering and support have typically been done on the side in support of the ASCR-driven research products. DOE funding priorities have been slow to recognize that good software engineering, the kind that ensures research investments have more adoption and longevity, requires significant resources. Based upon our experiences, we have prepared this response to highlight the concerns and issues we believe to be important as DOE ASCR considers its role in scientific software stewardship. We believe that role is important and will require a significant investment of new funding to legitimately support the technologies past and future DOE ASCR investments have and will produce to facilitate their uptake and adoption in the broader scientific computing community. Following a summary of our involvement in scientific software development, the remainder our response is organized around the nine topics specifically identified in the RFI.

The lifetime of offshore field developments can be as long as 20 to 30 years. Operators must have continuous focus on their assets throughout the field life. The monitoring of the operational data is therefore a key issue to assess integrity and to ensure a safe and optimal operation of the flexible risers. In co-operation with CorrOcean ASA, Technip-Coflexip has established a Pipe Integrity Management (PIM) system where relevant operational data, monitoring and inspection results are gathered into a database, treated and analyzed. This paper describes the PIM system recently implemented for Statoil ASA on the Åsgard field (North Sea). We present the database principle and the most relevant operational parameters monitored in operation. The findings of the operational data review are then discussed and compared with risers original design assumptions. Finally the paper presents the benefits to Operator of such a system. Introduction Most Operators operate flexible pipes without a dedicated system to follow up the integrity of the flexible pipes. Hence, any evaluation or assessment of the pipe conditions requires substantial collection of original design data, manufacturing data and not least operational data. Experience has shown than the gathering of relevant operational data can be difficult already after some years in service. In addition, especially for long field life (15 to 20 years), the operational conditions may sometimes differ compared to assumed original design basis data. Without a good knowledge of the operational riser history, the flexible riser integrity assessment can become a difficult exercise, leading for example to either operational limitations or even riser change-out, which could have been avoided or delayed if good monitoring data had been available. Statoil's Åsgard field development consists of three main floating production and storage units linked by 36 flexible risers. The operating conditions are characterized by high temperatures (130 °Celsius design temperature) and high pressures (up 500 bars injection pressure) combined with large gas volumes. Hence, the riser monitoring system is extremely important for a safe and reliable operation of those risers. Statoil decided in 2001 to develop a reliable Pipe Integrity Management (PIM) system with Technip-Coflexip. The PIM ystem is not intended to substitute the Operator's day to day monitoring system, which remains the responsibility of the Operator's organization. The overall objectives of the PIM system can be summarized as follow:A transfer of the relevant operational data from the Operator's database into the PIM must be done regularly to ensure continuous verification and backup of the riser data, and verification of the risers. Annual transfer is chosen for Åsgard. The frequency may be increased if required.

New generation of health care IT systems are collecting and storing more and more data of patients. Useful knowledge can be extracted from the data in EMR or PHR to provide medical advises to patients, while through data analysis the result statistics can be used to support the scientific research. However, RDBMSs-based framework is not able to support the requirements of massive health care data storage, management and analysis. To solve the problem, this paper proposes a massive data management and analysis solution based on Hadoop to archive better performance, scalability and fault tolerance. The data management framework is presented. Besides, 2 different data analysis methods based on MapReduce and Hive are proposed. Experiment results of data upload, data query and data analysis show that the performance of the proposed framework is greatly improved, and a brief summary of the performance and the differences between 2 methods of MapReduce and Hive is also discussed.