Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction

Bilingual lexicon induction (BLI) from comparable data has become a common way of evaluating cross-lingual word embeddings (CWEs). These models have drawn much attention, mainly due to their availability for rare and low-resource language pairs. An alternative offers systems exploiting parallel data, such as popular neural machine translation systems (NMTSs), which are effective and yield state-of-the-art results. Despite the significant advancements in NMTSs, their effectiveness in the BLI task compared to the models using comparable data remains underexplored. In this paper, we provide a comparative study of the NMTS and CWE models evaluated on the BLI task and demonstrate the results across three diverse language pairs: distant (Estonian-English) and close (Estonian-Finnish) language pair and language pair with different scripts (Estonian-Russian). Our study reveals the differences, strengths, and limitations of both approaches. We show that while NMTSs achieve impressive results for languages with a great amount of training data available, CWEs emerge as a better option when faced less resources.

Bilingual lexicon induction is the task of inducing word translations from monolingual corpora in two languages. In this article we present the most comprehensive analysis of bilingual lexicon induction to date. We present experiments on a wide range of languages and data sizes. We examine translation into English from 25 foreign languages: Albanian, Azeri, Bengali, Bosnian, Bulgarian, Cebuano, Gujarati, Hindi, Hungarian, Indonesian, Latvian, Nepali, Romanian, Serbian, Slovak, Somali, Spanish, Swedish, Tamil, Telugu, Turkish, Ukrainian, Uzbek, Vietnamese, and Welsh. We analyze the behavior of bilingual lexicon induction on low-frequency words, rather than testing solely on high-frequency words, as previous research has done. Low-frequency words are more relevant to statistical machine translation, where systems typically lack translations of rare words that fall outside of their training data. We systematically explore a wide range of features and phenomena that affect the quality of the translations discovered by bilingual lexicon induction. We provide illustrative examples of the highest ranking translations for orthogonal signals of translation equivalence like contextual similarity and temporal similarity. We analyze the effects of frequency and burstiness, and the sizes of the seed bilingual dictionaries and the monolingual training corpora. Additionally, we introduce a novel discriminative approach to bilingual lexicon induction. Our discriminative model is capable of combining a wide variety of features that individually provide only weak indications of translation equivalence. When feature weights are discriminatively set, these signals produce dramatically higher translation quality than previous approaches that combined signals in an unsupervised fashion (e.g., using minimum reciprocal rank). We also directly compare our model's performance against a sophisticated generative approach, the matching canonical correlation analysis (MCCA) algorithm used by Haghighi et al. ( 2008 ). Our algorithm achieves an accuracy of 42% versus MCCA's 15%.

Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embeddings. Such methods critically rely on those embeddings having a similar structure, but it was recently shown that the separate training in different languages causes departures from this assumption. In this paper, we propose an alternative approach that does not have this limitation, while requiring a weak seed dictionary (e.g., a list of identical words) as the only form of supervision. Rather than aligning two fixed embedding spaces, our method works by fixing the target language embeddings, and learning a new set of embeddings for the source language that are aligned with them. To that end, we use an extension of skip-gram that leverages translated context words as anchor points, and incorporates self-learning and iterative restarts to reduce the dependency on the initial dictionary. Our approach outperforms conventional mapping methods on bilingual lexicon induction, and obtains competitive results in the downstream XNLI task.

We present InstaMap, an instance-based method for learning projection-based cross-lingual word embeddings. Unlike prior work, it deviates from learning a single global linear projection. InstaMap is a non-parametric model that learns a non-linear projection by iteratively: (1) finding a globally optimal rotation of the source embedding space relying on the Kabsch algorithm, and then (2) moving each point along an instance-specific translation vector estimated from the translation vectors of the point’s nearest neighbours in the training dictionary. We report performance gains with InstaMap over four representative state-of-the-art projection-based models on bilingual lexicon induction across a set of 28 diverse language pairs. We note prominent improvements, especially for more distant language pairs (i.e., languages with non-isomorphic monolingual spaces).

Bilingual lexicon induction, translating words from the source language to the target language, is a long-standing natural language processing task. Recent endeavors prove that it is promising to employ images as pivot to learn the lexicon induction without reliance on parallel corpora. However, these vision-based approaches simply associate words with entire images, which are constrained to translate concrete words and require object-centered images. We humans can understand words better when they are within a sentence with context. Therefore, in this paper, we propose to utilize images and their associated captions to address the limitations of previous approaches. We propose a multi-lingual caption model trained with different mono-lingual multimodal data to map words in different languages into joint spaces. Two types of word representation are induced from the multi-lingual caption model: linguistic features and localized visual features. The linguistic feature is learned from the sentence contexts with visual semantic constraints, which is beneficial to learn translation for words that are less visual-relevant. The localized visual feature is attended to the region in the image that correlates to the word, so that it alleviates the image restriction for salient visual representation. The two types of features are complementary for word translation. Experimental results on multiple language pairs demonstrate the effectiveness of our proposed method, which substantially outperforms previous vision-based approaches without using any parallel sentences or supervision of seed word pairs.

Bilingual lexicon induction focuses on learning word translation pairs, also known as bitexts, from monolingual corpora by establishing a mapping between the source and target embedding spaces. Despite recent advancements, bilingual lexicon induction is limited to inducing bitexts consisting of individual words, lacking the ability to handle semantics‐rich phrases. To bridge this gap and support downstream cross‐lingual tasks, it is practical to develop a method for bilingual phrase induction that extracts bilingual phrase pairs from monolingual corpora without relying on cross‐lingual knowledge. In this paper, the authors propose a novel phrase embedding training method based on the skip‐gram structure. Specifically, a local hard negative sampling strategy that utilises negative samples of central tokens in sliding windows to enhance phrase embedding learning is introduced. The proposed method achieves competitive or superior performance compared to baseline approaches, with exceptional results recorded for distant languages. Additionally, we develop a phrase representation learning method that leverages multilingual pre‐trained language models. These mPLMs‐based representations can be combined with the above‐mentioned static phrase embeddings to further improve the accuracy of the bilingual phrase induction task. We manually construct a dataset of bilingual phrase pairs and integrate it with MUSE to facilitate the bilingual phrase induction task.

Language modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across languages. We propose a bilingual attention language model (BALM) that simultaneously performs language modeling objective with a quasi-translation objective to model both the monolingual as well as the cross-lingual sequential dependency. The attention mechanism learns the bilingual context from a parallel corpus. BALM achieves state-of-the-art performance on the SEAME code-switch database by reducing the perplexity of 20.5% over the best-reported result. We also apply BALM in bilingual lexicon induction, and language normalization tasks to validate the idea.

Bilingual Lexicon Induction: Effortless Evaluation of Word Alignment Tools and Production of Resources for Improbable Language Pairs

Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations. Inspired by the global paradigm shift in NLP towards Large Language Models (LLMs), we examine the potential of the latest generation of LLMs for the development of bilingual lexicons. We ask the following research question: Is it possible to prompt and fine-tune multilingual LLMs (mLLMs) for BLI, and how does this approach compare against and complement current BLI approaches? To this end, we systematically study 1) zero-shot prompting for unsupervised BLI and 2) few-shot in-context prompting with a set of seed translation pairs, both without any LLM fine-tuning, as well as 3) standard BLI-oriented fine-tuning of smaller LLMs. We experiment with 18 open-source text-to-text mLLMs of different sizes (from 0.3B to 13B parameters) on two standard BLI benchmarks covering a range of typologically diverse languages. Our work is the first to demonstrate strong BLI capabilities of text-to-text mLLMs. The results reveal that few-shot prompting with in-context examples from nearest neighbours achieves the best performance, establishing new state-of-the-art BLI scores for many language pairs. We also conduct a series of in-depth analyses and ablation studies, providing more insights on BLI with (m)LLMs, also along with their limitations.

Recent embedding-based methods in bilingual lexicon induction show good results, but do not take advantage of orthographic features, such as edit distance, which can be helpful for pairs of related languages. This work extends embedding-based methods to incorporate these features, resulting in significant accuracy gains for related languages.

This paper describes the sourcing, processing, and application of parallel text data for Icelandic and Polish for the purpose of bilingual lexicon induction (BLI), demonstrating how a parallel corpus can be compiled for a low-to-medium resource language pair that has no available parallel data, by pivoting through a common language. We show the usefulness of the corpus by training and evaluating a machine translation (MT) model on the data. Iceland's linguistic landscape is evolving, with an increasing need for multilingual support due to the growing immigrant population. Polish, in particular, stands out as the language of the largest single minority in Iceland, underscoring the importance of this project.

Learning bilingual sentiment lexicon for online reviews

Analysis of a corpus of errors in pronouncing Proper Nouns used in English collected from the spontaneous speech of a sample of Arab informants showed that Arabic speakers have the following problems: (i) mispronouncing English vowels in Google, Moodle, Uber, Nixon, London; (ii) replacing consonants absent in L1 (p v) by their equivalents (bebsi, jafa); (iii) geminating consonants in city and country names (Peking; Venezuela, Minnesota); (iv); inserting a vowel in consonant clusters in Proper Nouns and acronyms (Zelinsky, Logansk, SNAS, GMC, ); (v) breaking words into two sub-words (Kasper+sky, Sky+pe); (vi) pronouncing words the way they are spelled (Nazi, Nike, Huawei, Hyundai, Wednesday); and (vii) transferring Arabic stress rules to English words (McDonald, Mayflower). Pronunciation errors made by Arab students are attributed to transfer from the native language, i.e., Modern Standard Arabic (MSA) or the local dialect, insufficient mastery of English pronunciation rules, phonics and phone-grapheme correspondences and lack of knowledge of the differences between English and Arabic phonology. To correct pronunciation problems, the study recommends that students watch pronunciation practice videos that focus on a single error or rule at a time; raising students’ awareness of English pronunciation rules, phonics, phoneme-grapheme correspondences and comparing English and Arabic vowels, consonants, syllable structure, pause, juncture and stress rules; providing direct pronunciation instruction, graded lessons and remedial exercises. Future research needs to investigate Arab students’ pronunciation problems in assimilation, elision, and others.

The goal of this paper is to explain the various asymmetries with regard to the (non-)use of definite articles in diverse languages by exploiting the distinction of semantic and pragmatic uniqueness as originally introduced by Lobner (Journal of Semantics 4: 279–326, 1985). I put forward the claim that typologically speaking, there are two kinds of such definite article splits. Both of them follow the scale of uniqueness Lobner (Journal of Semantics 28: 279–333, 2011), a concept hierarchy that is defined by the (in)variance of reference of nominal expressions. The first kind is a split such that the bottom segment of the scale is marked by the definite article, whereas the rest remains unmarked. The second kind of split is characterised by pragmatic and semantic uniqueness being morphosyntactically distinguished by different forms of the definite article, commonly a strong and a reduced form. I propose a few amendments to the scale of uniqueness so that the variation both between and within individual languages is captured in terms of spreading along the scale.

Studies in Language (SL) provides a forum for the discussion of issues in contemporary linguistics, with a particular focus on empirically well-grounded research in the functionalist tradition that recognizes the diversity and variability of human languages and of communication patterns, the historical dynamics of languages, and the embedding of language in both social practices and cognition. Studies in Language provides for three sections and types of contributions: Regular articles: Regular articles are expected to adopt a cross-linguistic or comparative perspective and to advance our understanding of human language as such, in terms of state-of-the-art theories, methods, and analytical models or frameworks. Regular articles on one individual language are published only to the extent that they make a contribution of general interest. News from the Field: SL welcomes short contributions that report on new discoveries in little-known or endangered languages, emphasizing description over theory and comparison. Contributions to this special section typically derive from original fieldwork and are expected to provide concise and well-substantiated analyses of linguistic phenomena that have not been noticed so far and for which the wider theoretical implications have yet to be established. Discussions: SL publishes short position papers that outline or take issue with general visions of the field or address current developments. Contributions to this section may lead to response articles in the same or in a subsequent issue and one (very short) reply by the original contributor. All submissions are subject to double-blind reviewing. SL publishes its articles Online First.