Connecting to Hadoop

This paper presents a new means of selecting quality data for mining multiple data sources. Traditional data-mining strategies obtain necessary data from internal and external data sources and pool all the data into a huge homogeneous dataset for discovery. In contrast, our data-mining strategy identifies quality data from (internal and external) data sources for a mining task. A framework is advocated for generating quality data. Experimental results demonstrate that application of this new data collecting technique can not only identify quality data, but can also efficiently reduce the amount of data that must be considered during mining.

In the process of data integration using ontologies it is important to manage data from external data sources in the same way as data stored in the Knowledge Base. In previous papers [], [] the way of inference from data stored in the Knowledge Base, using Knowledge Cartography idea has been presented. However, this solution requires loading all data to the Knowledge Base. The solution presented in this paper shows how the Knowledge Cartography can be used to infer from data stored in external data sources, without loading them to the Knowledge Base. The presented solution is to enrich each data source with an additional layer that allows managing data using signatures. The paper additionally describes the results of tests comparing times of inference when data are loaded to the Knowledge Base and when data are fetched on demand.

Trends in differences between births and surviving infants reported for immunization program planning and external data sources in Eastern and Southern Africa 2000–2013

First of four chapters covering data assembly, being i) observation (predictors); ii) performance (target); iii) target definition; iv) data merge. (1) Make a plan—finalize the what, where, who and how of data, especially as regards internal and external data sources and how subject-level data will be matched. (2) Gather—identify key fields and determine what data aggregates are required, set retention rules and obtain retrospective histories from external sources. (3) Reduce—identify predictors to be dropped or ignored, especially those un- or under-populated, unpredictive, proscribed, highly correlated and … those populated after the decision was made. (4) Cleanse—individual records that are out-of-scope, underpopulated, and duplicates. Mention is also made of the treatment of outliers for continuous targets and of inconsistencies within the data {e.g. varying codes by time or source}. (5) Check—ensure file contents are consistent with expectations.

Objective. The COVID-19 pandemic created an unprecedented strain on the health care system, and administrators had to make many critical decisions to respond appropriately. This study sought to understand how health care administrators used data and information for decision making during the first 6 mo of the COVID-19 pandemic. Materials and Methods. We conducted semistructured interviews with administrators across University of Florida (UF) Health. We performed an inductive thematic analysis of the transcripts. Results. Four themes emerged from the interviews: 1) common types of health systems or hospital operations data; 2) public health and other external data sources; 3) data interaction, integration, and exchange; and 4) novelty and evolution in data, information, or tools used over time. Participants illustrated the organizational, public health, and regional information they considered essential (e.g., hospital census, community positivity rate, etc.). Participants named specific challenges they faced due to data quality and timeliness. Participants elaborated on the necessity of data integration, validation, and coordination across different boundaries (e.g., different hospital systems in the same metro areas, public health agencies at the local, state, and federal level, etc.). Participants indicated that even within the first 6 mo of the COVID-19 pandemic, the data and tools used for making critical decisions changed. Discussion. While existing medical informatics infrastructure can facilitate decision making in pandemic response, data may not always be readily available in a usable format. Interoperable infrastructure and data standardization across multiple health systems would help provide more reliable and timely information for decision making. Conclusion. Our findings contribute to future discussions of improving data infrastructure and developing harmonized data standards needed to facilitate critical decisions at multiple health care system levels.HighlightsThe study revealed common health systems or hospital operations data and information used in decision making during the first 6 mo of the COVID-19 pandemic.Participants described commonly used internal data sources, such as resource and financial reports and dashboards, and external data sources, such as federal, state, and local public health data.Participants described challenges including poor timeliness and limited local relevance of external data as well as poor integration of data sources within and across organizational boundaries.Results suggest the need for continued integration and standardization of health data to support health care administrative decision making during pandemics or other emergencies.

Today's software systems are riddled with security vulnerabilities that invite attack. We envisage a secure software design process at the architectural level, in which the security requirements are adequate, thus enabling appropriate security controls to be implemented to mitigate known threats and vulnerabilities. How can we ensure that the security requirements are adequate? In this paper, we tackle this question by focusing on how external online data sources for vulnerabilities, attack patterns, threat intelligence, and other security information can be leveraged, using Natural Language Processing (NLP), to produce a report to assist designers in validating the adequacy of the security requirements. This validation is done by determining which requirements map to known threats (identified from the external data), which requirements may be extraneous, and which threats may need a closer look to identify new requirements. We first describe the availability and nature of the external data, followed by how we employ NLP to process the data and produce the report. We include an illustrative example of our approach.

The National Basketball Association (NBA; also referred to as "the league") has established a centralized, audited electronic medical record system that has been linked with external sources to provide a platform for robust research and to allow the NBA to conduct player health and safety reviews. The system is customized and maintained by the NBA and individual teams as part of the employment records for each player and is deployed uniformly across all 30 teams in the league, thereby allowing for standardized data on injuries, illnesses, and player participation in NBA games and practices. The electronic medical record data are enriched by linkage with other league external data sources that provide additional information about injuries, players, game and practice participation, and movement. These data linkages allow for the assessment of potential injury trends, development of injury-prevention programs, and rule changes, with the ultimate goal of improving player health and wellness. The purpose of this article is to describe this NBA injury database, including the details of data collection, data linkages with external data sources, and activities related to reporter training and data quality improvement.

The distributed Data Retrieval (DR) model consists of k peers connected by a complete peer-to-peer communication network, and a trusted external data source that stores an array X of n bits (n ≫ k). Up to β k of the peers might fail in any execution (for β ∈ [0, 1)). Peers can obtain the information either by inexpensive messages passed among themselves or through expensive queries to the source array X. In the DR model, we focus on designing protocols that minimize the number of queries performed by any nonfaulty peer (a measure referred to as the query complexity) while maximizing the resiliency parameter β. The Download problem requires each nonfaulty peer to correctly learn the entire array X. Earlier work on this problem focused on synchronous communication networks and established several deterministic and randomized upper and lower bounds. Our work is the first to extend the study of distributed data retrieval to asynchronous communication networks. We address the Download problem under both the Byzantine and crash failure models. We present query-optimal deterministic solutions in an asynchronous model that can tolerate any fixed fraction β < 1 of crash faults. In the Byzantine failure model, it is known that deterministic protocols incur a query complexity of Ω(n) per peer, even under synchrony. We extend this lower bound to randomized protocols in the asynchronous model for β ≥ 1/2, and further show that for β < 1/2, a randomized protocol exists with near-optimal query complexity.

There are many ways to store data in the modern-day including blob storage such as Amazon web service's S3 or Microsoft Azure's Blob Storage, NoSQL database such as columnar, wide column, object orientated, key-value databases and relational databases. When it comes to maintaining relationship-based data, SQL has been the industry standard for decades. In current times COVID-19 has plagued the world as one of the world's fastest growing pandemics. To fight against the spread, vaccines have been administered to millions of individuals. All this information about vaccine administration has to be stored and potentially made available as a "vaccine passport". We propose to use a full stack solution of MySQL, a relational database, and Python to implement a proof-of-concept solution to store, create, update, and maintain information records about the shots administered and who the shots were administered to as a vaccine passport.

Whenever I run an AZ-204 (Azure Developer Fundamentals) training session, I always start with Azure Storage. Storage is not necessarily sexy, and it’s not the first thing you often think about when getting ready to study for an Azure exam. However, storage is the essential backbone to pretty much anything you do in Azure.

ABSTRACTBI solutions are constantly being developed to support decision-making at various organisational levels. These solutions facilitate the compilation, aggregation and summarisation of large volumes of data. Consequently, the business value created by these systems is increasing as they sustain more and more advanced analytics, ranging from descriptive analytics, to predictive analytics, to prescriptive analytics. However, most organisations work primarily with internal data. Despite many references in the literature to the value hidden in external data, details on how such data can be used are scarce. In this paper, we present the results of an extensive action case study at a public waste management company. The results illustrate how external data from several external data sources, integrated into an up-and-running BI solution, are used jointly to allow for descriptive and predictive analytics, as well as prescriptive analytics. In addition, details of these analytical values are given and related to organisational benefits.

Network entity landmark is the key foundation of IP geolocaiton which plays an important role in network security. Integrating multi-source landmarks to generate a landmark database with high IP coverage and high location accuracy is an important solution for improving the IP geolocation effect. However, the city-level landmarks provide the city name while street-level landmarks provide the latitude and longitude where they located. Owning to their inconsistent format, the state-of-art fusion algorithm cannot effectively integrate the two types of data. Hence, this paper proposes Lusion, a multi-source landmark fusion algorithm. We first extend the IP addresses in the landmark data sources, then model the location data of the two types of landmarks using the landmark location mixture model, and finally use the expectation -maximization algorithm to estimate the location of the landmarks. The simulation experiments on 25 landmark data sources show that the algorithm can effectively integrate the city-level and street-level landmarks from different data sources, and have a significantly better performance than the original data sources in the location accuracy. Furthermore, we evaluate Lusion on real-world datasets, which consists of 7 city-level and 3 street-level landmark data sources, by locating 100 IP addresses in Hong Kong and Zhengzhou respectively. The geolocation results show that Lusion increased the city-level accuracy by at least 8 percentage points compared with the original data sources, and reduces the geolocation error from 3.42 km to 2.92 km based on the best original landmark data set.

We present COLUMBA, an integrated database of protein annotations. COLUMBA is centered around proteins whose structure has been resolved and adds as much annotations as possible to those proteins, describing their proper-ties such as function, sequence, classification, textual description, participation in pathways, etc. Annotations are extracted from seven (soon eleven) external data sources. In this paper we describe the motivation for building COLUMBA, its integrational architecture and the software tools we developed for the integrated data sources and keeping COLUMBA up-to-date. We put special focus on two aspects: First, COLUMBA does not try to remove redundancies and overlaps in data sources, but views each data source as a proper dimension describing a protein. We explain the advantages of this approach compared to a tighter semantic integration as pursued in many other projects. Second, we highlight our current investigations regarding the quality of data in COLUMBA by identification of hot spots of poor data quality.

There is an increasing demand for utilization of external data, such as historical study data and patient registry data, to augment the control group in a randomized controlled trial. While such a study design could reduce the time and cost, how to maintain the study validity and integrity is one major statistical challenge that needs to be carefully addressed. We discuss a study design quality process to enhance the study validity and integrity when using this approach. The discussed quality process is tailored to the confirmatory study using a 2-stage design with an emphasis on the interaction process among stakeholders. In an example, the quality process covers a 2-step assessment of the similarity in patient characteristics between the current study and the external data source, and between the treatment and augmented control groups.

In clinical trials, it is often valuable to borrow information from external data sources. Unfortunately, when the external data are fully or partially incompatible with the current trial data, type I error rates can be highly inflated under traditional blanket discounting schemes such as power priors, commensurate priors, and meta-analytic predictive priors. However, such inflation of the probability of a false positive can be necessary, as the alternative is to have an underpowered study. For clinical trials with time-to-event (TTE) outcomes, this problem is exacerbated since many observations are censored. In this paper, we develop the latent exchangeability prior for TTE data. We also present a novel framework to borrow information about the treatment effect between groups as well as incorporate information from external controls. Simulation results suggest that, although efficiency gains can be achieved by borrowing information among external controls, operating characteristics in general can be quite poor under a lack of exchangeability. We apply our approach to a real clinical trial in second-line metastatic colorectal cancer.