SEMANTIC SIMILARITY ANALYSIS USING TRANSFORMER-BASED SENTENCE EMBEDDINGS

In natural language processing (NLP) tasks, computing semantic textual similarity (STS) is crucial for capturing nuanced semantic differences in text. Traditional word vector methods, such as Word2Vec and GloVe, as well as deep learning models like BERT, face limitations in handling context dependency and polysemy and present challenges in computational resources and real-time processing. To address these issues, this paper introduces two novel methods. First, a sentence embedding generation method based on Kullback–Leibler Divergence (KLD) optimization is proposed, which enhances semantic differentiation between sentence vectors, thereby improving the accuracy of textual similarity computation. Second, this study proposes a framework incorporating RoBERTa knowledge distillation, which integrates the deep semantic insights of the RoBERTa model with prior methodologies to enhance sentence embeddings while preserving computational efficiency. Additionally, the study extends its contributions to sentiment analysis tasks by leveraging the enhanced embeddings for classification. The sentiment analysis experiments, conducted using a Stochastic Gradient Descent (SGD) classifier on the ACL IMDB dataset, demonstrate the effectiveness of the proposed methods, achieving high precision, recall, and F1 score metrics. To further augment model accuracy and efficacy, a feature selection approach is introduced, specifically through the Dynamic Principal Component Selection (DPCS) algorithm. The DPCS method autonomously identifies and prioritizes critical features, thus enriching the expressive capacity of sentence vectors and significantly advancing the accuracy of similarity computations. Experimental results demonstrate that our method outperforms existing methods in semantic similarity computation on the SemEval-2016 dataset. When evaluated using cosine similarity of average vectors, our model achieved a Pearson correlation coefficient (τ) of 0.470, a Spearman correlation coefficient (ρ) of 0.481, and a mean absolute error (MAE) of 2.100. Compared to traditional methods such as Word2Vec, GloVe, and FastText, our method significantly enhances similarity computation accuracy. Using TF-IDF-weighted cosine similarity evaluation, our model achieved a τ of 0.528, ρ of 0.518, and an MAE of 1.343. Additionally, in the cosine similarity assessment leveraging the Dynamic Principal Component Smoothing (DPCS) algorithm, our model achieved a τ of 0.530, ρ of 0.518, and an MAE of 1.320, further demonstrating the method’s effectiveness and precision in handling semantic similarity. These results indicate that our proposed method has high relevance and low error in semantic textual similarity tasks, thereby better capturing subtle semantic differences between texts.

Semantic similarity, the task of determining whether two sentences convey the same meaning, is central to applications such as paraphrase detection, semantic search, and question answering. Despite the widespread adoption of transformer-based models for this task, their performance is influenced by both the choice of similarity measure and BERT (bert-base-nli-mean-tokens), RoBERTa (all-roberta-large-v1), and MPNet (all-mpnet-base-v2) on the Microsoft Research Paraphrase Corpus (MRPC). Sentence embeddings were compared using cosine similarity, dot product, Manhattan distance, and Euclidean distance, with thresholds optimized for accuracy, balanced accuracy, and F1-score. Results indicate a consistent advantage for MPNet, which achieved the highest accuracy (75.6%), balanced accuracy (71.0%), and F1-score (0.836) when paired with cosine similarity at an optimized threshold of 0.671. BERT and RoBERTa performed competitively but exhibited greater sensitivity to the choice of Similarity metric, with BERT notably underperforming when using cosine similarity compared to Manhattan or Euclidean distance. Optimal thresholds varied widely (0.334–0.867), underscoring the difficulty of establishing a single, generalizable cut-off for paraphrase classification. These findings highlight the value of fine-tuning of both Similarity metrics and thresholds alongside model selection, offering practical guidance for designing high-accuracy semantic similarity systems in real-world NLP applications.

In this study, we proposed a novel scale abbreviation method based on sentence embeddings and compared it to two established automatic scale abbreviation techniques. Scale abbreviation methods typically rely on administering the full scale to a large representative sample, which is often impractical in certain settings. Our approach leverages the semantic similarity among the items to select abbreviated versions of scales without requiring response data, offering a practical alternative for scale development. We found that the sentence embedding method performs comparably to the data-driven scale abbreviation approaches in terms of model fit, measurement accuracy, and ability estimates. In addition, our results reveal a moderate negative correlation between item discrimination parameters and semantic similarity indices, suggesting that semantically unique items may result in a higher discrimination power. This supports the notion that semantic features can be predictive of psychometric properties. However, this relationship was not observed for reverse-scored items, which may require further investigation. Overall, our findings suggest that the sentence embedding approach offers a promising solution for scale abbreviation, particularly in situations where large sample sizes are unavailable, and may eventually serve as an alternative to traditional data-driven methods.

Automatic service discovery in heterogeneous environment is becoming one of the challenging problems for applications in semantic web, wireless sensor networks, etc. It is mainly due to the lack of accurate semantic similarity assessment between profile attributes of user request and web services. Generally, lexical semantic resources consist of corpus and domain knowledge. To improve similarity measures in terms of accuracy, various hybrid methods have been proposed to either integrate different semantic resources or combine various similarity methods based on a single resource. In this work, we propose a novel approach which combines vector similarity metrics in a continuous vector space to evaluate semantic similarity between concepts. This approach takes advantage of both corpus and knowledge base by constructing diverse vector space models. Specifically, we use differential evolutionary (DE) algorithm which is an powerful population-based stochastic search strategy for obtaining optimal value of the combination. Our approach has been validated against a variety of vector-based similarity approaches on multiple benchmark datasets. The empirical results demonstrate that our approach outperforms the state-of-the-art approaches. The results also indicate the continuous vectors are efficient for evaluating semantic similarity, since they have outstanding expressiveness to latent semantic features of words. Moreover, the robustness of our approach is presented by the steady measure results under different hyper-parameters of neural network. Keywords-differential evolutionary; semantic similarity; continuous vector space; vector similarity metrics

Conversational negation often behaves differently from negation as a logical operator: when rejecting a state of affairs, it does not present all members of the complement set as equally plausible alternatives, but it rather suggests some of them as more plausible than others (e.g., "This is not a dog, it is a wolf/*screwdriver"). Entities that are semantically similar to a negated entity tend to be judged as better alternatives (Kruszewski etal., 2016). In fact, Kruszewski etal. (2016) show that the cosine similarity scores between the distributional semantics representations of a negated noun and its potential alternatives are highly correlated with the negated noun-alternatives human plausibility ratings. In a series of cloze tasks, we show that negation likewise restricts the production of plausible alternatives to similar entities. Furthermore, completions to negative sentences appear to be even more restricted than completions to an affirmative conjunctive context, hinting at a peculiarity ofnegation.

Transformer-based models have recently shown success in representation learning on graph-structured data beyond natural language processing and computer vision. However, the success is limited to small-scale graphs due to the drawbacks of full dot-product attention on graphs such as the quadratic complexity with respect to the number of nodes and message aggregation from enormous irrelevant nodes. To address these issues, we propose Deformable Graph Transformer (DGT) that performs sparse attention via dynamically selected relevant nodes for efficiently handling large-scale graphs with a linear complexity in the number of nodes. Specifically, our framework first constructs multiple node sequences with various criteria to consider both structural and semantic proximity. Then, combining with our learnable Katz Positional Encodings, the sparse attention is applied to the node sequences for learning node representations with a significantly reduced computational cost. Extensive experiments demonstrate that our DGT achieves superior performance on 7 graph benchmark datasets with 2.5 $\sim$∼ 449 times less computational cost compared to transformer-based graph models with full attention.

The assessment of semantic similarity between lexical terms plays a critical part in semantic-oriented applications for natural language processing and cognitive science. The optimization of calculation models is still a challenging issue for improving the performance of similarity measurement. In this paper, we investigate WordNet-based measures including distance-based, information-based, feature-based and hybrid. Among them, the distance-based measures are considered to have the lowest computational complexity due to simple distance calculation. However, most of existing works ignore the meronymy relation between concepts and the non-uniformity of path distances caused by various semantic relations, in which path distances are simply determined by conceptual hyponymy relation. To solve this problem, we propose a novel model to calculate the path distance between concepts, and also propose a similarity measure which nonlinearly transforms the distance to semantic similarity. In the proposed model, we assign different weights in accordance with various relations to edges that link different concepts. On basis of the distance model, we use five structure properties of WordNet for similarity measurement, which consist of multiple meanings, multiple inheritance, link type, depth and local density. Our similarity measure is compared against state-of-the-art WordNet-based measures on M&C dataset, R&G dataset and WS-353 dataset. According to experiment results, the proposed measure in this work outperforms others in terms of both Pearson and Spearman correlation coefficients, which indicates the effectiveness of our distance model. Besides, we construct six additional benchmarks to prove that the proposed measure maintains stable performance.

Multilingual sentence embeddings capture rich semantic information not only for measuring similarity between texts but also for catering to a broad range of downstream cross-lingual NLP tasks. State-of-the-art multilingual sentence embedding models require large parallel corpora to learn efficiently, which con- fines the scope of these models. In this paper, we propose a novel sentence embedding framework based on an unsupervised loss function for generating effective multilingual sentence embeddings, eliminating the need for parallel corpora. We capture semantic similarity and relatedness between sentences using a multi-task loss function for training a dual encoder model mapping different languages onto the same vector space. We demonstrate the efficacy of an unsupervised as well as a weakly supervised variant of our framework on STS, BUCC and Tatoeba benchmark tasks. The proposed unsupervised sentence embedding framework outperforms even supervised state-of-the-art methods for certain under-resourced languages on the Tatoeba dataset and on a monolingual benchmark. Further, we show enhanced zero-shot learning capabilities for more than 30 languages, with the model being trained on only 13 languages. Our model can be extended to a wide range of languages from any language family, as it overcomes the requirement of parallel corpora for training.

At present, the existing anomaly segmentation algorithm directly uses the abnormal score of the trained network output, such as the cosine similarity score. However, due to the uncertainty of the edge, the edge pixels or pixels near the edge are less similar to the class prototype. It shows more false negatives at the target edge during inferring, which will greatly affect the performance of the anomalous segmentation. This paper proposed to solve the problem of more false negatives in target edges. First, we need to calculate the cosine similarity between the input pixel and the class prototype. The cosine similarity calculation image was performed using morphological edge detection to extract the false negative edges, features with high non-edge similarity were screened by the edge-aware pooling layer, the filtered features are used to update the edges part. Finally, the updated cosine similarity calculation image is treated with expansion smoothing. The experiments presented were conducted in the StreetHazards Dataset, the FPR95 was 19.6%, AUROC rate was 92.5%, AUPR was 12.5%. The method in this paper is simple and the network does not require retraining, and the precision is close to SOTA.

In a biomedical literature search, the link between a query and a document is often not established, because they use different terms to refer to the same concept. Distributional word embeddings are frequently used for detecting related words by computing the cosine similarity between them. However, previous research has not established either the best embedding methods for detecting synonyms among related word pairs or how effective such methods may be. In this study, we first create the BioSearchSyn set, a manually annotated set of synonyms, to assess and compare 3 widely used word-embedding methods (word2vec, fastText, and GloVe) in their ability to detect synonyms among related pairs of words. We demonstrate the shortcomings of the cosine similarity score between word embeddings for this task: the same scores have very different meanings for the different methods. To address the problem, we propose utilizing pool adjacent violators (PAV), an isotonic regression algorithm, to transform a cosine similarity into a probability of 2 words being synonyms. Experimental results using the BioSearchSyn set as a gold standard reveal which embedding methods have the best performance in identifying synonym pairs. The BioSearchSyn set also allows converting cosine similarity scores into probabilities, which provides a uniform interpretation of the synonymy score over different methods. We introduced the BioSearchSyn corpus of 1000 term pairs, which allowed us to identify the best embedding method for detecting synonymy for biomedical search. Using the proposed method, we created PubTermVariants2.0: a large, automatically extracted set of synonym pairs that have augmented PubMed searches since the spring of 2019.

The kendallknight package introduces an efficient implementation of Kendall's correlation coefficient computation, significantly improving the processing time for large datasets without sacrificing accuracy. The kendallknight package, following Knight (1966) and posterior literature, reduces the time complexity resulting in drastic reductions in computation time, transforming operations that would take minutes or hours into milliseconds or minutes, while maintaining precision and correctly handling edge cases and errors. The package is particularly advantageous in econometric and statistical contexts where rapid and accurate calculation of Kendall's correlation coefficient is desirable. Benchmarks demonstrate substantial performance gains over the Base R implementation, especially for large datasets.

Feature-based approaches to semantic similarity assessment of concepts using Wikipedia

n proposed work, we successfully implemented a news text summarization system using Natural Language Processing (NLP) techniques and the Latent Semantic Analysis (LSA) algorithm. The purpose of our project was to extract important information from a large volume of news articles and present it in a concise and easily understandable manner. To achieve this, we utilized the LSA algorithm, which is known for its ability to capture the underlying semantic structure of text. LSA employs a mathematical model to analyse relationships between words in a document, creating a semantic representation where words with similar contexts are grouped together in a vector space. The LSA-based summarization process involved several steps. First, we pre-processed the news articles by removing stop words, punctuation, and other non-relevant elements. Then, we constructed a term-document matrix, where rows represented words and columns represented documents, with matrix values representing word frequencies. Next, we applied Singular Value Decomposition (SVD) to the term-document matrix. SVD helped reduce the matrix's dimensionality by identifying the most important latent semantic concepts. This resulted in a lower-dimensional representation that captured the essential information. Finally, we identified the most important sentences in the news articles by measuring the cosine similarity between each sentence and the summary. Sentences with the highest cosine similarity scores were selected as summary sentences. The proposed system demonstrated the effectiveness of the LSA algorithm for news text summarization. By capturing the semantic structure of the text, it generated summaries that allowed users to understand the key points of a news article quickly and easily. Our implementation had practical applications for content recommendation systems, news aggregation platforms, and personalized news feeds. However, it is important to acknowledge the limitations of the LSA algorithm. It may struggle with handling idiomatic expressions and can be sensitive to the quality of the input data. These considerations highlight the need for ongoing research and development to enhance the performance and robustness of news text summarization systems.

For the multi-enterprise collaborative modeling environment, the semantic conflicts of concept in the merger from the local model into the overall model or in the integration from the lower model into the upper model, the semantic-based multi-enterprise heterogeneous model fusion method is proposed. Moreover, the semantic similarity among the model instances is analyzed from the various levels, and based on the semantic similarity, a series of rules of the model merging are proposed, and then model integration is completed. Finally, the similarity matching tool is developed to realize based on semantic similarity analysis.

Modern manufacturing paradigms have incorporated Prognostics and Health Management (PHM) to implement data-driven methods for fault detection, failure prediction, and assessment of system health. The maintenance operation has similarly benefitted from these advancements, and predictive maintenance is now being used across the industry. Despite these developments, most of the approaches in maintenance rely on numerical data from sensors and field devices for any sort of analysis. Text data from Maintenance Work Orders (MWOs) contain some of the most crucial information pertaining to the functioning of systems and components, but are still regarded as ‘black holes’, i.e., they store valuable data without being used in decision-making. The analysis of this data can help save time and costs in maintenance. While Natural Language Processing (NLP) methods have been very successful in understanding and examining text data from non-technical sources, progress in the analysis of technical text data has been limited. Non-technical text data are usually structured and consist of standardized vocabularies allowing the use of out-of-the-box language processing methods in their analysis. On the other hand, records from MWOs are often semi-structured or unstructured; and consist of complicated terminologies, technical jargon, and industry-specific abbreviations. Deploying traditional NLP to such data can result in an imprecise and flawed analysis which can be very costly. Owing to these challenges, we propose a Technical Language Processing (TLP) framework for PHM. To illustrate its capabilities, we use text data from MWOs of aircraft to address two scenarios. First, we predict corrective actions for new maintenance problems by comparing them with existing problems using syntactic and semantic textual similarity matching and evaluate the results with cosine similarity scores. In the second scenario, we identify and extract the most dominant topics and salient terms from the data using Latent Dirichlet Allocation (LDA). Using the results, we are able to successfully link maintenance problems to standardized maintenance codes used in the aviation industry.