Duplicate Question Pair Detection Using Sentence BERT and Machine Learning

Abstract

Translate Article

Identifying duplicate questions is essential for maintaining quality and reducing redundancy on question-answering platforms like Quora. While it is relatively easy for humans to detect semantic similarity between questions, automating this task poses a challenge due to the diversity in language expression. With the advancement of Natural Language Processing and deep learning, it has become possible to capture deeper semantic meaning in text. This study proposes a duplicate question detection method that leverages Sentence-BERT embeddings to understand semantic similarity between question pairs. The proposed method is evaluated on a sampled subset of the Quora Question Pairs dataset comprising 30,000 instances. Various features such as cosine similarity, absolute embedding differences, length differences, and common word counts are engineered from these embeddings. These features are then used to train multiple machine learning classifiers, including XGBoost, Random Forest, SVM, and Logistic Regression. To improve performance, an ensemble approach using majority voting is employed. This method effectively combines the power of deep semantic representation and classical machine learning to enhance prediction accuracy.