Cross-Language Information Retrieval based on category matching between language versions of a web directory

In Cross-Language Information Retrieval (CLIR) process, the translation effects have a direct impact o n the accuracy of follow-up retrieval results. In diction ary-based approach, we are dealing with the words t hat have more than one meaning which can decrease the retrieval performance if the query translation retur n an incorrect translations. These issues need to be ove rcome using efficient technique. In this study we p roposed a Cross-Language Information Retrieval (CLIR) method based on domain ontology using Quran concepts for disambiguating translation of the query and to improve the dictionary-based query translation. For experimentation, we use Quran ontology written in E nglish and Malay languages as a bilingual parallelcorpora and Quran concepts as a resource for cross- language query translation along with dictionary-ba sed translation. For evaluation, we measure the perform ance of three IR systems. IR 1 is natural language query IR, IR 2 is natural language query CLIR based on dictionary (as a Baseline) and IR 3 is the retrieval of this research proposed method using Mean Average Precision (MAP) and average precision at 11 points of recall. The experimental result shows that our prop osed method brings significant improvement in retri eval accuracy for English document collections, but defi cient for Malay document collections. The proposed CLIR method can obtain query expansion effect and improve retrieval performance in certain language.

In this paper, we propose a cross-language information retrieval (CLIR) method based on estimating for domains of the query using hierarchic structures of Web directories. To get the most appropriate translation of the queries, we utilize the Web directories written in many different languages as multilingual corpus for disambiguating translation of the query and for estimating the domain of search results using hierarchic structures of Web directories. From experimental evaluations, we found that there is an advantage in retrieval accuracy using our proposal for disambiguating translation in CLIR system. We found that it is effective to restrict to target fields of the query using lower level merged categories in order to acquire suited translation of the query.

A major challenge in Cross-Language Information Retrieval (CLIR) is the adoption of translation knowledge in retrieval models, as it affects the term weighting which is known to highly impact the retrieval performance. In this paper, we present an analytical study of using translation knowledge in CLIR. In particular, by adopting axiomatic analysis framework, we formulate the impacts of translation knowledge on document ranking as constraints that any cross-language retrieval model should satisfy. We then consider the state-of-the-art CLIR methods and check whether they satisfy these constraints. Finally, we show through empirical evaluation that violating one of the constraints harms the retrieval performance significantly which calls for further investigation.

This report is a summary of the accomplishments of the 'Scalable Solutions for Processing and Searching Very Large Document Collections' LDRD, which ran from FY08 through FY10. Our goal was to investigate scalable text analysis; specifically, methods for information retrieval and visualization that could scale to extremely large document collections. Towards that end, we designed, implemented, and demonstrated a scalable framework for text analysis - ParaText - as a major project deliverable. Further, we demonstrated the benefits of using visual analysis in text analysis algorithm development, improved performance of heterogeneous ensemble models in data classification problems, and the advantages of information theoretic methods in user analysis and interpretation in cross language information retrieval. The project involved 5 members of the technical staff and 3 summer interns (including one who worked two summers). It resulted in a total of 14 publications, 3 new software libraries (2 open source and 1 internal to Sandia), several new end-user software applications, and over 20 presentations. Several follow-on projects have already begun or will start in FY11, with additional projects currently in proposal.

Proper name transliteration, the pronunciation based translation of a proper name, is important to many multilingual natural language processing task, such as Statistical Machine Translation (SMT) and Cross Lingual Information Retrieval (CLIR). This task is extremely challenging due to the pronunciation difference between the source and target language. A given proper name can lead to many different transliterations. In the past, research efforts had demonstrated a 30-50% error using top-1 reference for transliteration. This error leads to performance degradation for many applications. In this paper, a novel approach to verify a given proper name transliteration pair using a discrete variant Hidden Markov Model (HMM) alignment is proposed. The state emission probabilities are derived from SMT phrase tables. The proposed method yields an Equal Error Rate (EER) of 3.73% on a 300 matched and 1000 unmatched name pairs test set. By comparison, the commonly used SMT framework yields 6.5% EER under the best configuration. The widely used edit distance approach has an EER of 22%. Our new method achieves high accuracy and low complexity, and provides an alternative for name transliteration in CLIR and other cross lingual natural language applications such as word alignment and machine translation.

Paraphrase identification is a crucial task in natural language understanding, especially in cross-language information retrieval. Nowadays, Multi-Task Deep Neural Network (MT-DNN) has become a state-of-the-art method that brings outstanding results in paraphrase identification [1]. In this paper, our proposed method based on MT-DNN [2] to detect similarities between English and Vietnamese sentences, is proposed. We changed the shared layers of the original MT-DNN from original the BERT [3] to other pre-trained multi-language models such as M-BERT [3] or XLM-R [4] so that our model could work on cross-language (in our case, English and Vietnamese) information retrieval. We also added some tasks as improvements to gain better results. As a result, we gained 2.3% and 2.5% increase in evaluated accuracy and F1. The proposed method was also implemented on other language pairs such as English – German and English – French. With those implementations, we got a 1.0%/0.7% improvement for English – German and a 0.7%/0.5% increase for English – French.

Cross-lingual word embeddings display words from different languages in the same vector space. They provide reasoning about semantics, compare the meaning of words across languages and word meaning in multilingual contexts, necessary to bilingual lexicon induction, machine translation, and cross-lingual information retrieval. This paper proposes an efficient approach to learn bilingual transform mapping between monolingual word embeddings in language pairs. We choose ten different languages from three different language families and downloaded their last update Wikipedia dumps1 1. https://dumps.wikimedia.org. Then, with some pre-processing steps and using word2vec, we produce word embeddings for them. We select seven language pairs from chosen languages. Since the selected languages are relative, they have thousands of identical words with similar meanings. With these identical dictation words and word embedding models of each language, we create training, validation and, test sets for the language pairs. We then use a generative adversarial network (GAN) to learn the transform mapping between word embeddings of source and target languages. The average accuracy of our proposed method in all language pairs is 71.34%. The highest accuracy is achieved for the Turkish-Azerbaijani language pair with the accuracy 78.32%., which is noticeably higher than prior methods.

International communication and multitude of information in several languages require information retrieval systems that can cross language borders. Cross-language information retrieval (CLIR) is defined as the retrieval of documents in another language than the language of the request or query in Anurag Seetha, et al (2004). The language of request is the source language and the language of documents is the target language. Among several approaches to CLIR, we opted for the method of translating queries for English-Hindi language pair based on a bilingual dictionary. This paper describes our first phase experiments with the CLIR system.

Proper nouns are usually primary keys in a query. Their correct translation might be necessary to maintain a good retrieval performance in a cross language information retrieval (CLIR) system. However, dictionaries only include the most commonly used proper nouns, like major countries and capitals. As they are spelling variants of each other in most languages, using an approximate string matching technique against the target database index is the common approach taken to find the target language correspondents of the original query key. N-gram technique proved to be the most effective among other approximate string matching techniques. As we are dealing with an English-Arabic CLIR system which involves two languages of different alphabets, we decided to combine transliteration with the n-gram technique to generate the different spelling variants of out of vocabulary (OOV) words. We call this technique: Transliteration Ngram (TNG). One issue that arises with the Arabic language is that words that are spelled similarly can have different meanings depending on the context of the sentence. This is particularly true for proper names, which usually have a meaning if used as a verb or adjective. To further enhance our transliteration approach, we chose to use part of speech (POS) disambiguation to reduce the number of unrelated words from the set transliterations obtained using TNG.

Todays, The internet has become a source of multi-lingual content. Users are not aware of multiple languages, so the language diversity becomes a great barrier for world communication. Cross-Language Information Retrieval (CLIR) provides a solution for this language barrier, where a user can search in his native language and get the relevant information in the required language. Currently, distributed word vector representation has a trend in various Natural Language Processing (NLP) task. These word vectors are used to identify similar contextual words. In this paper, we analyze the effectiveness of word vectors across the languages in Hindi-English CLIR. Skip-Gram Model (SGM) is used to learn bi-lingual word vectors from sentence aligned comparable corpus. IBM model is used to align the source language and target language words from sentence aligned comparable corpus. Best target language translation is selected with the help of top-n word alignments and word vectors.

This paper presents a multipronged approach to predict early risk of mental health issues from user-generated content in social media. Supervised learning and information retrieval methods are used to estimate the risk of depression for a user given the content of its posts in reddit. The approach presented here was evaluated on the CLEF eRisk 2017 pilot task. We describe the details of five systems submitted to the task, and compare their performance. The comparisons show that combining information retrieval and machine learning methods gives the best results.

AcknowledgmentsIntroduction1. Suggestion, Hypnosis, and Crime: Robert Wiene's The Cabinet of Dr. Caligari (1920), by Stefan Andriopoulos2. Of Monsters and Magicians: Paul Wegener's The Golem: How He Came into the World (1920), by Noah Isenberg3. Movies, Money, and Mystique, by Christian Rogowski4. No End to Nosferatu (1922), by Thomas Elsaesser5. Fritz Lang's Dr. Mabuse, the Gambler (1922): Grand Enunciator of the Weimar Era, by Tom Gunning6. Who Gets the Last Laugh? Old Age and Generational Change in F. W. Murnau's The Last Laugh (1924), by Sabine Hake7. Inflation and Devaluation: Gender, Space, and Economics in G. W. Pabst's The Joyless Street (1925), by Sara F. Hall8. Tradition as Intellectual Montage: F. W. Murnau's Faust (1926), by Matt Erlin9. Metropolis (1927): City, Cinema, Modernity, by Anton Kaes10. Berlin, Symphony of a Great City (1927): City, Image, Sound, by Nora M. Alter11. Surface Sheen and Charged Bodies: Louise Brooks as Lulu in Pandora's Box (1929), by Margaret McCarthy12. The Bearable Lightness of Being: People on Sunday (1930), by Lutz Koepnick13. National Cinemas / International Film Culture: The Blue Angel (1930) in Multiple Language Versions, by Patrice Petro14. Coming Out of the Uniform: Political and Sexual Emancipation in Leontine Sagan's M dchen in Uniform (1931), by Richard W. McCormick15. Fritz Lang's M (1931): An Open Case, by Todd Herzog16. Whose Revolution? The Subject of Kuhle Wampe (1932), by Marc SilbermanFilmographyContributorsIndex

The task of semantic code search is to retrieve code snippets from a source code corpus based on an information need expressed in natural language. The semantic gap between natural language and programming languages has for long been regarded as one of the most significant obstacles to the effectiveness of keyword-based information retrieval (IR) methods. It is a common assumption that “traditional” bag-of-words IR methods are poorly suited for semantic code search: our work empirically investigates this assumption. Specifically, we examine the effectiveness of two traditional IR methods, namely BM25 and RM3, on the CodeSearchNet Corpus, which consists of natural language queries paired with relevant code snippets. We find that the two keyword-based methods outperform several pre-BERT neural models. We also compare several code-specific data pre-processing strategies and find that specialized tokenization improves effectiveness.

As increasing numbers of non‐English resources have become available on the Web, the interesting and important issue of how Web users can retrieve documents in different languages has arisen. Cross‐language information retrieval (CLIR), the study of retrieving information in one language by queries expressed in another language, is a promising approach to the problem. Cross‐language information retrieval has attracted much attention in recent years. Most research systems have achieved satisfactory performance on standard Text REtrieval Conference (TREC) collections such as news articles, but CLIR techniques have not been widely studied and evaluated for applications such as Web portals. In this article, the authors present their research in developing and evaluating a multilingual English–Chinese Web portal that incorporates various CLIR techniques for use in the business domain. A dictionary‐based approach was adopted and combines phrasal translation, co‐occurrence analysis, and pre‐ and posttranslation query expansion. The portal was evaluated by domain experts, using a set of queries in both English and Chinese. The experimental results showed that co‐occurrence‐based phrasal translation achieved a 74.6% improvement in precision over simple word‐by‐word translation. When used together, pre‐ and posttranslation query expansion improved the performance slightly, achieving a 78.0% improvement over the baseline word‐by‐word translation approach. In general, applying CLIR techniques in Web applications shows promise.

This paper discusses the translation style and methods adopted by Sobi FM 101.9 Ilorin broadcasters to find equivalent versions for some selected English lexis and structure, translated into the Yoruba language. The main objective of this paper is to analyze the relevance of translation and trace the etymological foundation of the lexis and structure which compel radio broadcasters to use self-lexical coinage and neology. The work was carried out within the purview of Newmark levels of translation (level of naturalness, Referential, cohesive level) to trace the content and function of the lexis and structure. The theory adopted for this work is the knowledge gap theory; the main thrust of this theory is to bridge the language gap that exists in media through the use of the native language. Data for this research were gathered through the elicitation of a recorded news item from Sobi 101.9 FM Ilorin. Our research findings show that most translations from the source language (English) to the target language (Yorùbá) by broadcasters are self-invented, which could either suit the original version in the target language or not. It was observed that radio broadcasters adopt suitable translation methods to retain the content of messages broadcast in a bilingual program. We conclude our work by saying categorically that broad knowledge of the source language helps the productivity of a translator to transmit the message to the audience in the language without any vacuum or gap, and that radio broadcasters adopt any translation method in as much as the audience comprehension of the intended messages are the same with the target language.