Techniques, Applications, and Issues in Mining Large-Scale Text Databases

For information retrieval and text-mining, a robust scalable framework is required to represent the information extracted from documents and enable visualization and query of such information. One very widely used model is the vector space model which is based on the bag-of-words approach. However, it suffers from the fact that it loses important information about the original text, such as information about the order of the terms in the text or about the frontiers between sentences or paragraphs. In this paper, we propose a graph-based text representation, which is capable of capturing (i) Term order (ii) Term frequency (iii) Term co-occurrence (iv) Term context in documents. We also apply the graph model into our text mining task, which is to discover unapparent associations between two and more concepts (e.g. individuals) from a large text corpus. Counterterrorism corpus is used to evaluate the performance of various retrieval models, which demonstrates feasibility and effectiveness of graphic text representation in information retrieval and text mining.

Information explosion and availability of information in various forms has changed the shape of information centres and nature of information profession and professionals. Information profession and professionals have been impacted by the exponentially increasing volumes of information available — as well as with changing attitudes and behaviour of information seeker toward electronic resources. It is very difficult to find the required pieces of information in the bundled of scattered information. The task becomes more challenging, when we find that over 90% of the information available is in unstructured and semi-structured forms, which is very difficult to search. Here text mining has come as a tool to help Information professionals to find the relevant information and deliver to its users. Text mining is used as a technology for analyzing large volumes of structured/unstructured textual documents. Text mining has very high knowledge and commercial values. The aim of Text mining is generally to strengthen decision making and internal operations processes of any organisation and generation of new domain of knowledge. These technologies help to increase the utilization of Knowledge Management (KM) systems and pro-actively help information professionals to improve their competencies and thus productivity of the organization. This article discusses the basic concept of text mining, its framework and text mining products and tools. It also discusses the benefits and challenges of text mining and examines the role of Information Professionals in Text Mining.

Creating and critiquing explanations of phenomena is a significant goal of many scientific disciplines and therefore also a learning goal of science education. A significant source of explanations is science textbooks; however, the large corpus of text in textbooks means that manual review of explanations by individual researchers is extremely time consuming. In this paper, we introduce a text-mining approach for identifying legitimate and illegitimate forms of teleological explanations in school physics and biology textbooks. An ongoing debate exists about the legitimacy of teleological explanations, that is, explanations which account for a phenomenon by reference to a final end, purpose, or goal. Until recently, researchers tended to view teleology as an illegitimate form of scientific explanation. Recent theoretical cases in biology and physics have emphasized that legitimate teleological explanations exist in both domains. Eight science textbooks used in England and internationally were analyzed for instances of teleological explanations. The analysis reveals the efficiency of the text-mining approach for automating the analysis of textbooks and its potential as a research approach in science education. In considering text mining as a research approach, we report terms that are likely to be associated with legitimate teleological explanations. We found that legitimate teleological accounts are used by textbook authors, and we present novel categorizations of these forms of explanation. We argue that text mining can be a useful approach in science education research and our findings suggest guidance for both textbook writers and teachers related to their selection of legitimate explanatory forms.

Knowledge documents are growing remarkably to serve the organization for information processing and various management tasks. Text mining is hard but important research topic in knowledge discovery where hidden information is extracted from unstructured and semi-structured data. Patents are rich knowledge source needed to be organized efficiently and conveniently. Patent documents are used for gathering business intelligence and identifying key trends in technology development. The main focus of this study is to propose a electrical patent classification framework based on Semantic Deep Learner (SDL). In this framework, initially key terms of the patent documents are extracted and represented using Vector Space Model (VSM), the importance of the key terms are weighted based up on their frequencies using TF-IDF. The semantic similarity between the key features is computed using cosine measure. Terms with higher correlations are synthesized into a smaller set of features. Finally the semantic deep learner is trained using the correlated features and accordingly patents are classified. The target output identifies the category of a patent document based on a hierarchical classification scheme of the International Patent Classification (IPC) standard. Our approach is new to the patent domain and shows some improvement in the classification accuracy when compared to the other state of art classifier.

Text Mining for Translational Bioinformatics

Recent progress of corpus and machine learning-based natural language processing methodologies have made it possible to handle large scale corpus with a quite high accuracy. The speaker is now involved in a project for constructing a large scale contemporary Japanese balanced corpus, aiming at constructing automatic annotation tools on various levels of natural language analyses. I will first introduce our activities on corpus based natural language analyzers for word dependency parsing and anaphora resolution and annotated corpus management environment. Then, I will explain recent natural language applications such as sentiment/opinion mining and knowledge extraction from a large scale text data like Weblogs.

The sharp increase of the amount of Internet Chinese text data has significantly prolonged the processing time of classification on these data. In order to solve this problem, this paper proposes and implements a parallel naive Bayes algorithm (PNBA) for Chinese text classification based on Spark, a parallel memory computing platform for big data. This algorithm has implemented parallel operation throughout the entire training and prediction process of naive Bayes classifier mainly by adopting the programming model of resilient distributed datasets (RDD). For comparison, a PNBA based on Hadoop is also implemented. The test results show that in the same computing environment and for the same text sets, the Spark PNBA is obviously superior to the Hadoop PNBA in terms of key indicators such as speedup ratio and scalability. Therefore, Spark-based parallel algorithms can better meet the requirement of large-scale Chinese text data mining.

With the advent of the big-data era, massive textual information stored in electronic and digital documents have become valuable resources for knowledge discovery in the fields of design and engineering. Ontology technologies and semantic networks have been widely applied with text mining techniques including Natural Language Processing (NLP) to extract structured knowledge associations from the large-scale unstructured textual data. However, most existing works mainly focus on how to construct the semantic networks by developing various text mining methods such as statistical approaches and semantic approaches, while few studies are found to focus on how to subsequently analyze and fully utilize the already well-established semantic networks. In this paper, a specific network analysis method is proposed to discover the implicit knowledge associations from the existing semantic network for improving knowledge discovery and design innovation. Pythagorean means are applied with Dijkstra’s shortest path algorithm to discover the implicit knowledge associations either around a single knowledge concept or between two concepts. Six criteria are established to evaluate and rank the correlation degree of the implicit associations. Two engineering case studies were conducted to illustrate the proposed knowledge discovery process, and the results showed the effectiveness of the retrieved implicit knowledge associations on helping providing relevant knowledge from various aspects, and provoking creative ideas for engineering innovation.

The services that digital libraries provide to users can be greatly enhanced by automatically gleaning certain kinds of information from the full text of the documents they contain. This paper reviews some recent work that applies novel techniques of machine learning(broadly interpreted) to extract information from plain text, and puts it in the context of digital library applications. We describe three areas: hierarchical phrase browsing, including efficient methods for inferringa phrase hierarchy from a large corpus of text; text mining using adaptive compression techniques, giving a new approach to generic entity extraction, word segmentation, and acronym extraction; and keyphrase extraction.KeywordsRecent WorkMachine LearningAlgorithmic LearnEfficient MethodFull TextThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The services that digital libraries provide to users can be greatly enhanced by automatically gleaning certain kinds of information from the full text of the documents they contain. This paper reviews some recent work that applies novel techniques of machine learning (broadly interpreted) to extract information from plain text, and puts it in the context of digital library applications. We describe three areas: hierarchical phrase browsing, including efficient methods for inferring a phrase hierarchy from a large corpus of text; text mining using adaptive compression techniques, giving a new approach to generic entity extraction, word segmentation, and acronym extraction; and keyphrase extraction.KeywordsDigital LibraryText MiningToken TypeLearn StructurePlain TextThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Innovations in science and technology boost-up the generation of an immense volume of digital data in the form of text, audio, video, images and paved the way for a new era called Big Data. A majority of this data is generated as a semi-structured and unstructured format poses a challenge for Natural Language Processing (NLP) and text analytics. This chapter seeks to layout the text mining perspective of big data analytics with an emphasis on applications in biomedicine and healthcare. The chapter aims to provide a better understanding of areas such as Big Data, Text Mining, Data Mining, Natural Language Processing and Machine Learning and its inter linkage. The chapter illustrates phases and task of text mining in big data scope. Finally, the chapter ends with a description of the two application areas biomedicine and healthcare where text mining using big data analytics will be used.

This study uses LDA topic model to cluster and refine topics of research content in the field of ginseng health care.LDA (Latent Dirichlet Allocation) topic model is a probabilistic statistics-based text mining method, which is able to automatically discover potential topic distributions from large-scale text data. By setting the appropriate number of topics, the model assigns words in the text to different topics, thus realizing the classification and summarization of the research content. The evolutionary trend of each theme is analyzed in combination with the time dimension, and hot themes are identified. The study mines out 6 themes: Chinese medicine health care and disease prevention of ginseng health care, functional research and standard management of ginseng health food, detection and quality evaluation of ginsenoside components, development and market analysis of ginseng health care products, laboratory research and application of ginseng extract, and optimization of the extraction process of ginseng health care functions. Based on the combination of the development trend of the hot topics, future research can be innovative in the development and market analysis of ginseng health products.

QuantNet is an integrated web-based environment consisting of different types of statistics-related documents and program codes. Its goal is creating reproducibility and offering a platform for sharing validated knowledge native to the social web. To increase the information retrieval (IR) efficiency there is a need for incorporating semantic information. Three text mining models will be examined: vector space model (VSM), generalized VSM (GVSM) and latent semantic analysis (LSA). The LSA has been successfully used for IR purposes as a technique for capturing semantic relations between terms and inserting them into the similarity measure between documents. Our results show that different model configurations allow adapted similarity-based document clustering and knowledge discovery. In particular, different LSA configurations together with hierarchical clustering reveal good results under M3 evaluation. QuantNet and the corresponding Data-Driven Documents (D3) based visualization can be found and applied under quantlet. The driving technology behind it is Q3-D3-LSA, which is the combination of “GitHub API based QuantNet Mining infrastructure in R”, LSA and D3 implementation.

Text mining based on semantic annotations is increasingly used to support semantic search and knowledge discovery in a number of biomedical applications. Recently, the extraction of complex events based on rich semantic metadata has been used for customized text mining applications such as query-answering systems, advanced searching, etc. Semantic annotations, named entities, facts, associations, events: all enrich document collections and facilitate knowledge discovery

Scientists and engineers have had little understanding of dynamics and properties of human "everyday life" despite its familiarity. Although in scientific fields such as quantum theory and cosmology, standard models exist to explain and generate most phenomena, nothing yet exists which might represent a standard model of everyday life. Recent development of technologies for sensing physical phenomena in total space using ubiquitous sensors, technologies for sensing worldwide social phenomena using Internet technology, and modeling technologies for constructing a computational system based on the large-scale sensory and text data gained using these sensing technologies implies that everyday life is becoming a new target of computing. This paper discusses a unique methodology for everyday life computing with which we can develop the foundations of computational intelligence by tightly coupling with everyday life. In particular, this paper takes asp the issue of children injury prevention as a real problem to be tackled with computational intelligence technology, presenting a methodology for everyday life computing through describing some attempts at sensing and modeling of everyday life for injury prevention and will also outline some future prospects in this field