Content-based Information Retrieval Techniques Based on Grid Computing: A Review

Grid technology is widely used in recent years, it has solved the real sharing of resources, make each node of unified command and use of resources, the information grid is on the basis of computational grid, using data mining. Information fusion and search engine technology and building are advantageous for the collection and sharing of grid resources. The goal to create a build on OS and Web is based on the new generation of Internet information platform. In this platform, the distribution of the information processing is the collaboration and intelligent. Focus crawler sets according to the target subject. In the form of intelligent collection topics come from the Web page, and then adopt the method of machine learning. The information retrieval on the collected information intelligent need process and analyze. Finally, the effective retrieval way is to meet the demand of the user's information retrieval.

The wide exploitation of new techniques and systems for generating, collecting and storing data has made available growing volumes of information. Large quantities of such information are stored as free texts. The lack of explicit structure in free text is a major issue in the categorization of such kind of data for more effective and efficient information retrieval, search and filtering. The abundance of structured data is problematic too. Several databases are available, that contain data of the same type. Unfortunately, they often conform to different schemas, which avoids the unified management of even structured information. The Entity Resolution process plays a fundamental role in the context of information integration and management, aimed to infer a uniform and common structure from various large-scale data collections, with which to suitably organize, match and consolidate the information of the individual repositories into one data set. De-duplication is a key step of the Entity Resolution process, whose goal is discovering duplicates within the integrated data, i.e., different tuples that, as a matter of facts, refer to the same real-world entity. This attenuates the redundancy of the integrated data and, also, enables more effective information handling and knowledge extraction through a unified access to reconciled and de-duplicated data. Duplicate detection is an active research area that benefits from contributions from diverse research fields, such as, machine learning, data mining and knowledge discovery, databases as well as information retrieval and extraction. This chapter presents an overview of research on data de-duplication, with the goal of providing a general understanding and useful references to fundamental concepts concerning the recognition of similarities in very large data collections. For this purpose, a variety of state-of-the-art approaches to de-duplication is reviewed. The discussion of the state-of-the-art conforms to a taxonomy that, at the highest level, divides the existing approaches into two broad classes, i.e., unsupervised and supervised approaches. Both classes are further divided into sub-classes according to the common peculiarities of the involved approaches. The strengths and weaknesses of each group of approaches are presented. Meaningful research developments to further advance the current state-of-the-art are covered as well.

Indexing is an important process in Information Retrieval (IR) systems. It forms the core functionality of the IR process since it is the first step in IR and assists in efficient information retrieval. Indexing reduces the documents to the informative terms contained in them. It provides a mapping from the terms to the respective documents containing them. Once effective index has been built for the collection of documents, retrieval process is simplified. Indexing proceeds at four stages namely content specification, tokenization of documents, processing of document terms, and index building. The index can be stored in the form of different data structures namely direct index, document index, lexicon and inverted index. Index can be built by applying different algorithms or schemes such as single-pass in-memory indexing, blocked-indexing, etc. This paper explains the indexing process with the various data structures and algorithms used for indexing and finally analyses the different indexing approaches with respect to time, memory usage and mean average precision.

Learning to rank has recently emerged as an attractive technique dedicated to the optimization of results rankings and based on machine learning techniques. Several search engine applications are using this technique to train their ranking model. Theoretically, the problem of ranking is to predict an ordering over a set of pairs query-document. However, the enormously large size of the Web documents makes it generally impossible for the users to find their desired information by surfing the Web. As a consequence, effective and efficient information retrieval (IR) has become more important and also IR system (IRS) has become indispensable tool that can allow the user to access only to the information he deems relevant. Currently, we have proposed two pairwise learning to rank algorithms which combine active and semi-supervised learning to reduce the labeling effort for DR. In this paper, we propose to improve these semi-supervised active algorithms by using a list of pairs in selecting unlabeled data. We showed through different ranking measures that the algorithms proposed yielded into competitive results compared to SAL2R and ASSL2R on collections from the standard benchmark Letor.

The Web has become a large repository of documents (or pages) written in many different languages. In this context, traditional information retrieval (IR) techniques cannot be used whenever the user query and the documents being retrieved are in different languages. To address this problem, new cross‐language information retrieval (CLIR) techniques have been proposed. In this work, we describe a method for cross‐language retrieval of medical information. This method combines query terms and related medical concepts obtained automatically through a categorization procedure. The medical concepts are used to create a linguistic abstraction that allows retrieval of information in a language‐independent way, minimizing linguistic problems such as polysemy. To evaluate our method, we carried out experiments using the OHSUMED test collection, whose documents are written in English, with queries expressed in Portuguese, Spanish, and French. The results indicate that our cross‐language retrieval method is as effective as a standard vector space model algorithm operating on queries and documents in the same language. Further, our results are better than previous results in the literature.

The intention of this paper was to determine usability of KOHA OPAC in information retrieval with regard to usability testing by undergraduate users at Technical University of Kenya Library (TU-KL) in Nairobi County. Usability testing can be realized by way of assigning tasks to a representative of the user population in order to determine whether users can be able to achieve the desired goal of retrieving information. To a great extent, success will depend on how users perform the tasks assigned to them. This study conducted usability test on actual users of KOHA OPAC at Technical University of Kenya Library. Qualitative data was collected using the think aloud method where study participants were required to verbalize their thought process as they performed search and retrieval tasks using KOHA OPAC. The number of errors made, time taken to complete search and retrieval activities were coded, recorded and analysed during usability testing. This helped the researcher to determine whether KOHA OPAC is easy to use and whether it saves time for the user. Undergraduate students for the study were selected using a stratified random sampling technique. The population of the study (n = 382) was determined using a simplified formula developed by Yamane. The study initiated a usability test which presented search scenarios where participants were invited to participate in searching and retrieving information resources using KOHA OPAC. The results of the usability test of KOHA OPAC at TU-KL showed that users were slow at accomplishing search tasks, they took long duration to access TU-KL website that hosts KOHA OPAC and there was inefficiency in accessing library materials. This study concluded that usability challenges may have an impact on users' overall satisfaction with KOHA OPAC and information retrieval effectiveness. In order to enhance user experience and boost the effectiveness of information retrieval, the study recommends improvement of KOHA OPAC usability by revamping its interface and enhancing its features.

Question Answering (QA) is a specialized subfield within Information Retrieval (IR) that focuses on providing precise answers to user queries. Recent advances in Large Language Models (LLMs) have greatly improved the speed and accuracy of question-answering (QA) systems, leading to significantly enhanced capabilities and performance. This study introduces a multi-PDF querying system designed to support automated question answering for Vietnamese students, leveraging the capabilities of LLMs. To enhance the retrieval accuracy from natural language queries, we integrate four Vietnamese-language embedding models. These models are evaluated using a dataset comprising questions related to the academic regulations of the University of Da Nang - University of Science and Technology (DUT), reflecting real-life scenarios students commonly encounter. Experimental results reveal that the top-performing Vietnamese embedding model achieves a Mean Average Precision (mAP) of 0.840 and an F1-score of 0.959, demonstrating its high effectiveness in information retrieval. This study makes significant contributions to the integration of Vietnamese embedding models into automated question-answering systems, aiming to effectively support students in the university environment.

Performance measures are frequently used to evaluate user friendliness of a system. An equally important, but often overlooked factor is the users' attitudes towards a system. A prototype interface for information retrieval was developed for presenting engineering manuals online. It was tested on a representative sample of the intended end user community. We found that subjects' expectations were based on their experience with printed materials and other computer systems. Familiar search mechanisms (e.g., table of contents, index) were important for getting them started, even though they switched to other mechanisms as they gained more experience with the system. The fact that the index was more detailed than the one in the printed manual was seen by the subjects as critical for speedy and efficient information retrieval. Keyword search of the database was generally the preferred retrieval mechanism. However, some users preferred the index. The ‘Table of Contents’ which was a tree structured menu based system was found to be of limited use in the electronic medium, in contrast to the printed manual.

Evaluation is crucial in assessing the effectiveness of new information retrieval and human computer interaction techniques and systems. Relevance judgements are often performed by humans, which makes obtaining them expensive and time consuming. Consequently, relevance judgements are usually performed only on a subset of a given collection of data or experimental results with a focus on the top ranked documents. However, when assessing the performance of exploratory search systems, the diversity or subjective relevance of documents that the user was presented with over a search session are often of more importance than the relative ranking of top documents. In order to perform these types of assessment, all the documents in a given collection need to be judged for relevance. In this paper, we propose an approach based on topic modeling that can greatly accelerate document relevance judgment of an entire document collection with an expert assessor needing to mark only a small subset of documents from a given collection. Experimental results show a substantial overlap between relevance judgments compared to a human assessor.

The volume of web search data stored in search engine log is increasing and has become big search log data. The web search log has been the source of data for mining based on web document clustering techniques to improve the efficiency and effectiveness of information retrieval. In this paper Deep Learning Model Convolution Neural Network(CNN) is used in big web search log data mining to learn the semantic representation of a document. These semantic documents vectors are clustered using K-means to group relevant documents for effective web document clustering. Experiment was done on the data set of web search query and associated clicked URLs to measure the quality of clusters based on document semantic representation using Deep learning model CNN. The clusters analysis was performed based on WCSS(the sum of squared distances of documents samples to their closest cluster center) and decrease in the WCSS in comparison to TF.IDF keyword based clusters confirm the effectiveness of CNN in web search log mining for effective web document clustering.

Search Engine (SE) is the most used information retrieval tool in the present scenario. In spite of the huge involvement of users in search engines, their limited capabilities to understand the user context and emotions places high load on the user to maintain the search momentum. Thus the research is being done on the web search process including the users. The aim is to reduce the contextual and emotional mismatch between the SE's and users. Personalized user query processing, current knowledge updation on users, information retrieval effectiveness and user satisfaction are the major challenges to be addressed in this field. To meet the high level requirements of the users and to improve the intellectualization level of the search, various cognitive agent based SE models are proposed in the literature. The challenges of the cognitive agents in present search engine scenario are obtaining sufficient percepts from the environment, understanding it, gathering the knowledge and retrieving the information from the huge volume of data. Recent advancements in the field of semantic search engines, knowledge gathering techniques and information retrieval methods are discussed which can better the cognitive agent based search engines by addressing the challenges. Overall the goal is to enhance the search experience by fewer repetitions of queries and making it easier for the users to retrieve the highly relevant documents of their interest.

Information Retrieval (IR) systems aim to retrieve data that satisfies certain requirements and constitute an important service in many types of networks, including Delay/Disruption Tolerant Networks (DTNs). In current DTN based IR systems, the data that satisfies a query is assumed to be stored on a single node. Therefore, once a node receives a query in which it has the corresponding data, the query can be resolved completely. However, in scenarios where a query requires data from multiple nodes, these IR systems may fail. Henceforth, in this paper, we propose Distributed Data-Centric Information Retrieval (DDC-IR), a data centric IR system that supports all query types; e.g., continuous and complex. More importantly, it is designed specifically to operate in DTNs. It also incorporates a new packet, aka Query Reply Packet, that includes both a query and one or more replies. We show how this packet facilitates efficient query resolution and enables data centric routing. In addition, it uses caching so that nodes store popular queries that has the effect of speeding up query resolution. We have conducted an extensive simulation study to compare DDC-IR to state of the art IR systems using the popular Random Waypoint model and a trace-file containing student movements on a campus. The results show that DDC-IR is able to resolve 50 % more queries and has an 80 % lower buffer occupancy level than existing IR systems. We also tested DDC-IR in networks with varying sizes. For networks with 100 nodes, DDC-IR is able to resolve queries while current IR systems fail to resolve any queries. In particular, when the number of nodes increases, current IR systems fail to resolve any queries, whilst DDC-IR is able to resolve complex and continuous queries. The influence of the number of sub-queries on query resolution time is also studied. Specifically, when the number of sub-queries in a complex query increases from five to nine, DDC-IR uses 50 % more time to resolve a query. In comparison, prior IR systems fail to resolve any queries.

This paper presents a vector space model approach to representing documents and queries, which considers concepts instead of terms and uses WordNet as a light ontology. This representation reduces information redundancy with respect to conventional semantic expansion techniques. Experiments carried out on the MuchMore benchmark and on the TREC-7 and TREC-8 Ad-hoc collections demonstrate the effectiveness of the proposed approach.

Recall, the proportion of relevant documents retrieved, is an important measure of effectiveness in information retrieval, particularly in the legal, patent, and medical domains. Where document sets are too large for exhaustive relevance assessment, recall can be estimated by assessing a random sample of documents, but an indication of the reliability of this estimate is also required. In this article, we examine several methods for estimating two-tailed recall confidence intervals. We find that the normal approximation in current use provides poor coverage in many circumstances, even when adjusted to correct its inappropriate symmetry. Analytic and Bayesian methods based on the ratio of binomials are generally more accurate but are inaccurate on small populations. The method we recommend derives beta-binomial posteriors on retrieved and unretrieved yield, with fixed hyperparameters, and a Monte Carlo estimate of the posterior distribution of recall. We demonstrate that this method gives mean coverage at or near the nominal level, across several scenarios, while being balanced and stable. We offer advice on sampling design, including the allocation of assessments to the retrieved and unretrieved segments, and compare the proposed beta-binomial with the officially reported normal intervals for recent TREC Legal Track iterations.

Latent Dirichlet Allocation (LDA), is heavily cited in the machine learning literature, but its feasibility and effectiveness in information retrieval is mostly unknown. Learning to rank is useful for document retrieval, it uses feature vector to rank, but there is no feature about document topic. Our paper combines LDA and learning to rank, adds a topic feature into the feature vector of learning to rank algorithm. And the experiment results show that the topic feature has improved the rank result effectively.