Cross-Language Information Retrieval by Domain Restriction Using Web Directory Structure

Cross language information retrieval (CLIR) is a retrieval process in which the user fires queries in one language to retrieve information from another (different) language. The diversity of information and language barriers are the serious issues for communication and cultural exchange across the world. To solve such barriers, Cross language information retrieval system, are nowadays in strong demand. CLIR is a subset of Information Retrieval (IR) system. Information Retrieval deals with finding useful information from a large collection of unstructured, structured and semi-structured data to a user query where the query is a set of keywords. Information Retrieval can be classified into different classes such as Monolingual information retrieval, Bi-Lingual Information Retrieval, Multilingual information retrieval and Cross language information retrieval. This paper focuses on the various IR variants and techniques used in CLIR system. Further, based on available literature, a number of challenges and issues in CLIR have been identified and discussed. It gives an overview of the advantages, limitations, tools available in CLIR research. It also describes new application areas of CLIR such as medical, multimedia, question answering system etc. The need for exploring and building more specialized information system that enable speakers of an Odia language to discover valuable information beyond linguistic and cultural barriers. This study is aimed at building an experimental CLIR system between one of the under-resourced language (i.e. Odia) and one of the most commonly used online language (i.e. English) in future.

Proper nouns are usually primary keys in a query. Their correct translation might be necessary to maintain a good retrieval performance in a cross language information retrieval (CLIR) system. However, dictionaries only include the most commonly used proper nouns, like major countries and capitals. As they are spelling variants of each other in most languages, using an approximate string matching technique against the target database index is the common approach taken to find the target language correspondents of the original query key. N-gram technique proved to be the most effective among other approximate string matching techniques. As we are dealing with an English-Arabic CLIR system which involves two languages of different alphabets, we decided to combine transliteration with the n-gram technique to generate the different spelling variants of out of vocabulary (OOV) words. We call this technique: Transliteration Ngram (TNG). One issue that arises with the Arabic language is that words that are spelled similarly can have different meanings depending on the context of the sentence. This is particularly true for proper names, which usually have a meaning if used as a verb or adjective. To further enhance our transliteration approach, we chose to use part of speech (POS) disambiguation to reduce the number of unrelated words from the set transliterations obtained using TNG.

Speech recognition has provided limited converted speech into text as query for cross language information retrieval (CLIR). Therefore, in this study, speech recognition to be integrated with cross language information retrieval system to investigate the Quran system results performance using semantic and stemmers as stemming semantic spoken query (SSSQ). The query however is based on the speech to be input and converted into text. Therefore, this study is conducted with the purposes to investigate the integration speech, semantic and stemmers approach against the queries and vice versa. Furthermore, it is also conducted to investigate the performance query based on total retrieve and relevant. The retrieval however, included the irrelevant documents because of the translation polysemy. Results from the experiments suggest that SSSQ is most important process in CLIR. It also found that semantic speech approach with stemmers contributes to better approach in using speech instead of keyboard to investigate its performance for Quran document results.

International communication and multitude of information in several languages require information retrieval systems that can cross language borders. Cross-language information retrieval (CLIR) is defined as the retrieval of documents in another language than the language of the request or query in Anurag Seetha, et al (2004). The language of request is the source language and the language of documents is the target language. Among several approaches to CLIR, we opted for the method of translating queries for English-Hindi language pair based on a bilingual dictionary. This paper describes our first phase experiments with the CLIR system.

Since the Web consists of documents in various domains or genres, the method for Cross-Language Information Retrieval (CLIR) of Web documents should be independent of a particular domain. In this paper, we propose a CLIR method which employs a Web directory provided in multiple language versions (such as Yahoo!). In the proposed method, feature terms are first extracted from Web documents for each category in the source and the target languages. Then, one or more corresponding categories in another language are determined beforehand by comparing similarities between categories across languages. Using these category pairs, we intend to resolve ambiguities of simple dictionary translation by narrowing the categories to be retrieved in the target language.

Sentence alignment using local and global information

The study of cross-lingual Information Retrieval Systems (IRSs) and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation MultiLingual Information Access (MLIA) systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of Cross Language Information Retrieval (CLIR) systems: statistical analyses to provide MLIA researchers with quantitative and more sophisticated analysis techniques; and graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the Cross-Language Evaluation Forum (CLEF) 2005 and 2006 campaigns.KeywordsInformation RetrievalMean Average PrecisionGraphical ToolBilingual ComparisonBilingual PerformanceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In Cross-Language Information Retrieval (CLIR) process, the translation effects have a direct impact o n the accuracy of follow-up retrieval results. In diction ary-based approach, we are dealing with the words t hat have more than one meaning which can decrease the retrieval performance if the query translation retur n an incorrect translations. These issues need to be ove rcome using efficient technique. In this study we p roposed a Cross-Language Information Retrieval (CLIR) method based on domain ontology using Quran concepts for disambiguating translation of the query and to improve the dictionary-based query translation. For experimentation, we use Quran ontology written in E nglish and Malay languages as a bilingual parallelcorpora and Quran concepts as a resource for cross- language query translation along with dictionary-ba sed translation. For evaluation, we measure the perform ance of three IR systems. IR 1 is natural language query IR, IR 2 is natural language query CLIR based on dictionary (as a Baseline) and IR 3 is the retrieval of this research proposed method using Mean Average Precision (MAP) and average precision at 11 points of recall. The experimental result shows that our prop osed method brings significant improvement in retri eval accuracy for English document collections, but defi cient for Malay document collections. The proposed CLIR method can obtain query expansion effect and improve retrieval performance in certain language.

This report is a summary of the accomplishments of the 'Scalable Solutions for Processing and Searching Very Large Document Collections' LDRD, which ran from FY08 through FY10. Our goal was to investigate scalable text analysis; specifically, methods for information retrieval and visualization that could scale to extremely large document collections. Towards that end, we designed, implemented, and demonstrated a scalable framework for text analysis - ParaText - as a major project deliverable. Further, we demonstrated the benefits of using visual analysis in text analysis algorithm development, improved performance of heterogeneous ensemble models in data classification problems, and the advantages of information theoretic methods in user analysis and interpretation in cross language information retrieval. The project involved 5 members of the technical staff and 3 summer interns (including one who worked two summers). It resulted in a total of 14 publications, 3 new software libraries (2 open source and 1 internal to Sandia), several new end-user software applications, and over 20 presentations. Several follow-on projects have already begun or will start in FY11, with additional projects currently in proposal.

Paraphrase identification is a crucial task in natural language understanding, especially in cross-language information retrieval. Nowadays, Multi-Task Deep Neural Network (MT-DNN) has become a state-of-the-art method that brings outstanding results in paraphrase identification [1]. In this paper, our proposed method based on MT-DNN [2] to detect similarities between English and Vietnamese sentences, is proposed. We changed the shared layers of the original MT-DNN from original the BERT [3] to other pre-trained multi-language models such as M-BERT [3] or XLM-R [4] so that our model could work on cross-language (in our case, English and Vietnamese) information retrieval. We also added some tasks as improvements to gain better results. As a result, we gained 2.3% and 2.5% increase in evaluated accuracy and F1. The proposed method was also implemented on other language pairs such as English – German and English – French. With those implementations, we got a 1.0%/0.7% improvement for English – German and a 0.7%/0.5% increase for English – French.

With the rapid expansion of multilingual digital resources, multilingual information retrieval (MLIR) has gained significant importance in information science. While fusing results from multiple retrieval systems can enhance MLIR performance, integrating too many systems leads to computational inefficiency and information redundancy, challenging the balance between effectiveness and cost. To address this, we propose an AI-driven Multilingual Retrieval Fusion (AI-MLRF) method, which incorporates advanced natural language processing techniques and a weighted greedy subset selection strategy to optimize the fusion process. AI-MLRF consists of four core modules: semantic performance evaluation using cross-lingual pre-trained models, diversity measurement based on contrastive learning, weighted greedy subset selection, and attention-based result fusion. Experimental evaluations on benchmark datasets demonstrate that AI-MLRF outperforms state-of-the-art fusion methods, showing substantial improvements in retrieval accuracy and adaptability to multilingual scenarios. The method provides a transparent and efficient solution for multilingual retrieval result fusion, with potential applications in cross-border collaboration and academic research.

We study the impact of translation resource scarcity on the performance of cross-language information retrieval (CLIR) systems. To do that, we develop a contrastive analysis framework that uses high-resource languages to simulate low-resource languages. In the framework, we focus on parallel translation corpora and aim to better understand the factors that impact CLIR performance. We argue that both low- and high-resource corpora are needed to develop that understanding. Hence, we take the approach of starting with a true low-resource language and systematically down-sampling a high-resource language to become an artificial low-resource language-the reverse perspective of existing research. We formalize the problem as the Resource Scarcity Simulation (RSS) problem. We model the problem with a family of set covering problems, formulate with integer linear programming, and prove that the problem is actually NP-hard. To this end, we provide two greedy algorithms with polynomial complexities. We compare and analyze our approach with alternate techniques using four high-resource languages (French, Italian, German, and Finnish) down-sampled to simulate two low-resource languages (Somali and Swahili). Our experimental results suggest that language families are important for the RSS problem. We simulate Somali with German, and Swahili with Finnish, achieving 98% and 97% on the similarity percentage in terms of CLIR performance, respectively.

Given a user question, the goal of a Question Answering (QA) system is to retrieve answers rather than full documents or even best-matching passages, as most Information Retrieval systems currently do. In this paper, we present BRUJA, a QA system for the management of multilingual collections. BRUJ rkstions (English, Spanish and French). The BRUJA architecture is not formed with three monolingual QA systems but instead uses English as Interlingua to make usual QA tasks such as question classifications and answer extractions. In addition, BRUJA uses Cross Language Information Retrieval (CLIR) techniques to retrieve relevant documents from a multilingual collection. On the one hand, we have more documents to find answers from but on the other hand, we are introducing noise into the system because of translations to the Interlingua (English) and the CLIR module. The question is whether the difficulty of managing three languages is worth it or whether a monolingual QA system delivers better results. We report on in-depth experimentation and demonstrate that our multilingual QA system gets better results than its monolingual counterpart whenever it uses good translation resources and, especially, CLIR techniques that are state-of-the-art.

Cross language information retrieval (CLIR) deals with the use of queries in one language to access documents in another. A popular CLIR approach is to translate the query into the language of the documents being retrieved. The simplest yet most effective method for query translation is bilingual dictionary approach. However, direct translation using bilingual dictionary prune two problems: proper names and compound words translations. In this study, a series of experiments were conducted to test and evaluate the effectiveness of the Malay-English CLIR using proper names and compound words translation. We believe by using concept-based indexing and translations, makes proper names and compound words translation possible. The best retrieval performance was obtained from the combination of query translation approach-select all translations listed in the dictionary, the alternative weighting scheme and proper names identification and translation.

Named Entity Recognition is a subtask in Information Extraction. In Named Entity Recognition (NER) we try to identify each of the words provided into some categories predefined by us. These categories or classes can be organization, name, time, place etc. Named Entity Recognition is a part of Natural Language Processing which is aims at making human and computer interactions more meaningful and efficient. NLP is nowadays being effectively being used in the area of automatic text summarization, cross language information retrieval, speech recognition and query named entity recognition. This paper focuses on Named entity recognition, how it is performed and a how apply NER to web search queries so that correct and more user personalized results are displayed by the web search engines..