A generic ranking function discovery framework by genetic programming for information retrieval

Discriminating meta-search: a framework for evaluation

We study the impact of translation resource scarcity on the performance of cross-language information retrieval (CLIR) systems. To do that, we develop a contrastive analysis framework that uses high-resource languages to simulate low-resource languages. In the framework, we focus on parallel translation corpora and aim to better understand the factors that impact CLIR performance. We argue that both low- and high-resource corpora are needed to develop that understanding. Hence, we take the approach of starting with a true low-resource language and systematically down-sampling a high-resource language to become an artificial low-resource language-the reverse perspective of existing research. We formalize the problem as the Resource Scarcity Simulation (RSS) problem. We model the problem with a family of set covering problems, formulate with integer linear programming, and prove that the problem is actually NP-hard. To this end, we provide two greedy algorithms with polynomial complexities. We compare and analyze our approach with alternate techniques using four high-resource languages (French, Italian, German, and Finnish) down-sampled to simulate two low-resource languages (Somali and Swahili). Our experimental results suggest that language families are important for the RSS problem. We simulate Somali with German, and Swahili with Finnish, achieving 98% and 97% on the similarity percentage in terms of CLIR performance, respectively.

The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. What makes a query difficult and how one may predict this is an active research area, focusing mainly on factors relating to the retrieval algorithm, to the properties of the retrieval data, or to statistical and linguistic features of the queries that may render them difficult. This work addresses query difficulty from a different angle, namely the users' own perspectives on query difficulty. Two research questions are asked: (1) Are users aware that the query they submit to an IR system may be difficult for the system to address? (2) Are users aware of specific features in their query (e.g., domain-specificity, vagueness) that may render their query difficult for an IR system to address? A study of 420 queries from a Web search engine query log that are pre-categorised as easy, medium, hard by TREC based on system performance, reveals an interesting finding: users do not seem to reliably assess which query might be difficult; however, their assessments of which query features might render queries difficult are notably more accurate. Following this, a formal approach is presented for synthesising the user-assessed causes of query difficulty through opinion fusion into an overall assessment of query difficulty. The resulting assessments of query difficulty are found to agree notably more to the TREC categories than the direct user assessments.

Extracting similar terms from multiple EMR-based semantic embeddings to support chart reviews

Glean: using syntactic information in document filtering

Retrieval of reliable relevant information is the major concern in medical information retrieval. Among all factors that affect the performance of retrieval system, ranking function is the most important factor. The retrieved documents from large document collection are arranged in the decreasing order of their relevance score by ranking function. A neuro-fuzzy system-based hybrid ranking function (HRF) is proposed in this paper. The proposed ranking function considers weight of document and query with respect to keyword as input features and gives relevance score between document and query as output. Experiments are performed on OHSUMED and PMC benchmark medical document corpus by using 15 experimental queries. The experimental results prove that the proposed HRF performs better when compared with fuzzy logic-based ranking function (FRF) and conventional statistical Euclidean distance-based ranking function (ERF) and cosine similarity-based ranking function (CRF) in terms of precision, recall and F-measure.

With the rapid evolution of smart home environment, the demand for spoken information retrieval (e.g., voice-activated FAQ retrieval) on information appliances is increasing. In spoken information retrieval, users’ spoken queries are converted into text queries using automatic speech recognition (ASR) engines. If top-1 results of the ASR engines are incorrect, the errors are propagated to information retrieval systems. If a document collection is a small set of sentences such as frequently asked questions (FAQs), the errors have additional effect on the performance of information retrieval systems. To improve the performance of such a sentence retrieval system, we propose a post-processing model of an ASR engine. The post-processing model consists of a re-ranking and a query term generation model. The re-ranking model rearranges top-n outputs of the ASR engines using the ranking support vector machine (Ranking SVM). The query term generation model extracts meaningful content words from the re-ranked queries based on term frequencies and query rankings. In the experiments, the re-ranking model improved the top-1 performance results of an underlying ASR engine with 4.4% higher precision and 6.4% higher recall rate. The query term generation model improved the performance results of an underlying information retrieval system with an accuracy 2.4% to 2.6% higher. Based on the experimental result, the proposed model revealed that it could improve the performance of a spoken sentence retrieval system in a restricted domain.

In recent years, both research and industry have shown an increasing interest in developing reliable information retrieval (IR) systems that can effectively address the growing demands of users worldwide. In spite of the relative success of IR systems in addressing the needs of users and even adapting to their environments, many problems remain unresolved. One main problem is lexical ambiguity which has negative impacts on the performance and reliability of IR systems. To date, lexical ambiguity has been one of the most frequently reported problems in the Arabic IR systems despite the development of different word sense disambiguation (WSD) techniques. This is largely attributed to the limitations of such techniques in addressing the issue of linguistic peculiarities. Hence, this study addresses these limitations by exploring the reasons for lexical ambiguity in IR applications in Arabic as one step towards reliable and practical solutions. For this purpose, the performances of six search engines Google, Bing, Baidu, Yahoo, Yandex, and Ask are evaluated. Results indicate that lexical ambiguities in Arabic IR applications are mainly due to the unique morphological and orthographic system of the Arabic language, in addition to its diglossia and the multiple colloquial dialects where sometimes mutual intelligibility is not achieved. For better disambiguation and IR performances in Arabic, this study proposes that clustering models based on supervised machine learning theory should be trained to address the morphological diversity of Arabic and its unique orthographic system. Search engines should also be adapted to the geographic location of the users in order to address the issue of vernacular dialects of Arabic. They should also be trained to automatically identify the different dialects. Finally, search engines should consider all varieties of Arabic and be able to interpret the queries regardless of the particular language adopted by the user.

Natural Language Processing (NLP) techniques are believed to hold the potential to assist bag-of-words Information Retrieval (IR) in terms of retrieval accuracy. In this paper, we report a natural language based IR approach where the common syntactic structures between documents and the query is regarded to as a query-dependent feature for documents. Specifically, a is proposed for query terms, which can be seen as a weight to model the degree of term's involvement in the common syntactic structures. This structural weight is used together with the TF-IDF weighting scheme, which results in a new ranking function. The accumulation of this structural weight of all the query terms in the new ranking function will be seen as a measure of how much a document and a query share the common syntactic structures. The experimental results show that by using this ranking function, significant improvements in the retrieval performance are achieved.

The information explosion across the Internet and elsewhere offers access to an increasing number of document collections. Retrieval system evaluation plays an important role in judging the efficiency and effectiveness of the retrieval process. Validating the performance of information retrieval systems provides assurance that a system as a whole is likely to meet its quantitative performance goals. In this paper, we describe procedure to calculate the response time early in the life cycle and simulate the performance metrics for an information retrieval system in different architectures namely, centralized and distributed. We consider different characterizations for the hardware resources' configurations and simulate the results for two different architectures of Information retrieval systems.

Collaborative Information Retrieval in an information-intensive domain

Recently, ranking data with respect to the intrinsic geometric structure (manifold ranking) has received considerable attentions, with encouraging performance in many applications in pattern recognition, information retrieval and recommendation systems. Most of the existing manifold ranking methods focus on learning a ranking function that varies smoothly along the data manifold. However, beyond smoothness, a desirable ranking function should vary monotonically along the geodesics of the data manifold, such that the ranking order along the geodesics is preserved. In this article, we aim to learn a ranking function that varies linearly and therefore monotonically along the geodesics of the data manifold. Recent theoretical work shows that the gradient field of a linear function on the manifold has to be a parallel vector field. Therefore, we propose a novel ranking algorithm on the data manifolds, called Parallel Field Ranking. Specifically, we try to learn a ranking function and a vector field simultaneously. We require the vector field to be close to the gradient field of the ranking function, and the vector field to be as parallel as possible. Moreover, we require the value of the ranking function at the query point to be the highest, and then decrease linearly along the manifold. Experimental results on both synthetic data and real data demonstrate the effectiveness of our proposed algorithm.

Ranking is likely the most important process of an information retrieval (IR) system that will be used to evaluate and measure the effectiveness of an IR system. This paper aims to produce the implementation of fuzzy logic controller of Mamdani-type fuzzy inference system for defining the ranking function by using the BM25 model in the Malay IR system that also includes the Malay Stemmer. The result of the ranking function then will be compared to the result of vector space model that is also applied in Malay IR system and be evaluated using relevant document by the hadith expert. The results showed that FBMIR has slightly outperformed vector space model on three topic set of query results such as 'Iman', 'Ilmu' and 'Wuduk' on the precision at rank 10 and the percentage of no relevant document in the top ten retrieved measures.

The performance of Information Retrieval Systems (IRSs) is usually measured using two different criteria, precision and recall. In such a way, the problem of tuning an IRS may be considered as a multi-objective optimization problem. In this contribution, we focus on the automatic learning of Boolean queries in IRSs by means of multi-objective evolutionary techniques. We present a comparative study of four multi-objective evolutionary optimization techniques of general-purpose (NSGA-II, SPEA2 and two MOGLS) to learn Boolean queries.

PurposeThe purpose of this paper is to introduce two new automatic methods for evaluating the performance of search engines. The reported study uses the methods to experimentally investigate which search engine among three popular search engines (Ask.com, Bing and Google) gives the best performance.Design/methodology/approachThe study assesses the performance of three search engines. For each one the weighted average of similarity degrees between its ranked result list and those of its metasearch engines is measured. Next these measures are compared to establish which search engine gives the best performance. To compute the similarity degree between the lists two measures called the “tendency degree” and “coverage degree” are introduced; the former assesses a search engine in terms of results presentation and the latter evaluates it in terms of retrieval effectiveness. The performance of the search engines is experimentally assessed based on the 50 topics of the 2002 TREC web track. The effectiveness of the methods is also compared with human‐based ones.FindingsGoogle outperformed the others, followed by Bing and Ask.com. Moreover significant degrees of consistency – 92.87 percent and 91.93 percent – were found between automatic and human‐based approaches.Practical implicationsThe findings of this work could help users to select a truly effective search engine. The results also provide motivation for the vendors of web search engines to improve their technology.Originality/valueThe paper focuses on two novel automatic methods to evaluate the performance of search engines and provides valuable experimental results on three popular ones.