An evaluation of classification models for question topic categorization

While diverse question answering (QA) datasets have been proposed and contributed significantly to the development of deep learning models for QA tasks, the existing datasets fall short in two aspects. First, we lack QA datasets covering complex questions that involve answers as well as the reasoning processes to get them. As a result, the state-of-the-art QA research on numerical reasoning still focuses on simple calculations and does not provide the mathematical expressions or evidence justifying the answers. Second, the QA community has contributed a lot of effort to improve the interpretability of QA models. However, they fail to explicitly show the reasoning process, such as the evidence order for reasoning and the interactions between different pieces of evidence. To address the above shortcoming, we introduce NOAHQA, a conversational and bilingual QA dataset with questions requiring numerical reasoning with compound mathematical expressions. With NOAHQA, we develop an interpretable reasoning graph as well as the appropriate evaluation metric to measure the answer quality. We evaluate the state-of-the-art QA models trained using existing QA datasets on NOAHQA and show that the best among them can only achieve 55.5 exact match scores, while the human performance is 89.7. We also present a new QA model for generating a reasoning graph where the reasoning graph metric still has a large gap compared with that of humans, eg, 28 scores.

Community question answering (CQA) services accumulate large amount of knowledge through the voluntary services of the community across the globe. In fact, CQA services gained much popularity recently compared to other Internet services in obtaining and exchanging information. Stack Overflow is an example of such a service that targets programmers and software developers. In general, most questions in Stack Overflow are usually ended up with an answer accepted by the askers. However, it is found that the number of unanswered or ignored questions has increased significantly in the past few years. Understanding the factors that contribute to questions being answered as well as questions remain ignored can help information seekers to improve the quality of their questions and increase their chances of getting answers from the community in Stack Overflow. In this study, we attempt to identify by data mining techniques the relevant features that will help predict the quality of questions, and validate the reliability of the features using some of the state-of-the-art classification algorithms. The features to be obtained have to be significant in the sense that they can help Stack Overflow to improve their existing CQA service in terms of user satisfaction in obtaining quality answers from their questions.

Objective: To use SMOTE to enhance class balance and compare the performance of different classification methods before and after applying SMOTE. Methods: The study used a dataset from Kaggle. Consisted of several health-related features linked to obesity prediction. Checking for class imbalance within the dataset affected initial model performance. SMOTE was applied to synthetically increase the representation of minority classes, reducing the class imbalance. It was conducted in two stages: 1. Training and testing the classification algorithms before applying SMOTE. 2. Training and testing the same models after applying SMOTE to enhance class balance. The performance of all models was evaluated based on metrics before and after the SMOTE application. Results: Models Logistic Regression and Naive Bayes struggled with low sensitivity and specificity, and KNN (k=5) showed poor specificity. Significant improvements were observed across all models after applying SMOTE. Logistic Regression, despite a decrease in accuracy(-8.8), sensitivity and specificity increased substantially(+56.7%), with balanced accuracy improving(+16.6%). Naive Bayes saw a modest accuracy increase(+2.3%), with sensitivity and specificity improving(+47.9%). The KNN classifier exhibited a transformative enhancement with sensitivity and specificity increasing(+96.0%) and balanced accuracy(+28.3%). Deep Learning showed a significant increase in sensitivity (+69.8%), balanced accuracy (+29.4%), and an improvement in precision and F1-score despite a slight decrease in specificity(-10.9%). Conclusion: The results demonstrate that while there might be slight trade-offs, the overall improvements in key metrics such as sensitivity, specificity, balanced accuracy, precision, and F1-score affirm the utility of SMOTE in enhancing model performance for imbalanced datasets

Nowadays, classifying sentiment from social media has been a strategic thing since people can express their feeling about something in an easy way and short text. Mining opinion from social media has become important because people are usually honest with their feeling on something. In our research, we tried to identify the problems of classifying sentiment from Indonesian social media. We identified that people tend to express their opinion in text while the emoticon is rarely used and sometimes misleading. We also identified that the Indonesian social media opinion can be classified not only to positive, negative, neutral and question but also to a special mix case between negative and question type. Basically there are two levels of problem: word level and sentence level. Word level problems include the usage of punctuation mark, the number usage to replace letter, misspelled word and the usage of nonstandard abbreviation. In sentence level, the problem is related with the sentiment type such as mentioned before. In our research, we built a sentiment classification system which includes several steps such as text preprocessing, feature extraction, and classification. The text preprocessing aims to transform the informal text into formal text. The word formalization method in that we use is the deletion of punctuation mark, the tokenization, conversion of number to letter, the reduction of repetition letter, and using corpus with Levensthein to formalize abbreviation. The sentence formalization method that we use is negation handling, sentiment relative, and affixes handling. Rule-based, SVM and Maximum Entropy are used as the classification algorithms with features of count of positive, negative, and question word in sentence and bigram. From our experimental result, the best classification method is SVM that yields 83.5% accuracy.

In this study, steady-state visual evoked potential based brain computer interface design and implementation was being carried out. A portable and affordable EEG device was used to obtain brain signals. Computer monitor were preferred as visual stimuli source. In offline and online experiment, for detection of target visual stimulus selected by user, the amplitude of the EEG signal components, which correspond visual stimulus frequencies, and correlation coefficient features were tested. In the classification stage the performance of different classification methods were compared and presented along tables.

Robust centroid based classification with minimum error rates for high dimension, low sample size data

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

In this paper, we propose a method for short text categorization using topic model and integrated classifier. To enrich the representation of short text, the Latent Dirichlet Allocation (LDA) model is used to extract latent topic information. While for classification, we combine two classifiers for achieving high reliability. Particularly, we train LDA models with variable number of topics using the Wikipedia corpus as external knowledge base, and extend labeled Web snippets by potential topics extracted by LDA. Then, the enriched representation of snippets are used to learn Maximum Entropy (MaxEnt) and support vector machine (SVM) classifiers separately. Finally, viewing that the most possible predicted result will appear in the top two candidates selected by MaxEnt classifier, we develop a novel scheme that if the gap between these candidates is large enough, the predicted result is considered to be reliable; otherwise, the SVM classifier will be integrated with MaxEnt classifier to make a comprehensive prediction. Experimental results show that our framework is effective and can outperform the state-of-the-art techniques.

Application of support vector machine (SVM) for large number of catalogs was studied, and the structure of optional SVM decision tree was introduced under the different background. Aiming at the defect recognition problem in X-ray inspection welding images and combining the basic theory of binary-tree, a binary-tree method of multi-class classification based on SVM was put forward. This method adopts 'one-against-all' classification algorithm by which binary-tree multi-classifier for welding defects based on SVM was established. According to the characteristics of defects, six parameters were chosen as feature parameters, and familiar defects in the weld were classified into 6 classes. 84 defect samples were used to experiments, and the results show that the classifier possesses simple, intuitionistic and practical algorithm, and small number of repeated training samples

This paper analyzes the performance of different classification methods for online activity recognition on mobile phones using the built-in accelerometers. First, we evaluate the performance of activity recognition using Naive Bayes and minimum distance classifiers and next we propose an improvement of Minimum Distance and K-Nearest Neighbor (KNN) classification algorithms called Clustered KNN. Clustered KNN eliminates the computational complexity of KNN by creating clusters, training sets for each activity. Classification is performed based on these reduced sets. We evaluate the performance of these classifiers on five test subjects for activities of walking, running, sitting and standing, and find that Naive Bayes provides not satisfactory results whereas Clustered KNN gives promising results.

Sentiment analysis is a crucial part of natural language processing (NLP) that categorizes text into positive, negative, or neutral sentiments. This technology helps businesses automatically interpret customer emotions, enabling informed decisions in marketing, product development, and customer support. Traditional machine learning models like Random Forest, Naive Bayes, and Support Vector Machine (SVM) have been commonly used for sentiment classification, especially effective for short and concise texts such as brief customer comments. SVM, in particular, performs well in classifying sentiments in short texts. However, the emergence of advanced large language models (LLMs) like GPT-4 has revolutionized sentiment analysis by capturing more nuanced emotional expressions. GPT-4 excels in understanding complex context, sarcasm, and subtle emotional tones within longer texts, offering multi-dimensional sentiment insights such as satisfaction, frustration, and uncertainty. This makes it more capable than traditional models when analyzing detailed customer reviews or conversations. Comparative studies reveal that while traditional models handle short texts efficiently, GPT-4 surpasses them in analyzing in-depth content with greater precision, recall, and F1 scores. GPT-4 also uniquely identifies mixed sentiments that simpler models often miss. In conclusion, although traditional machine learning approaches remain useful for straightforward sentiment analysis tasks, GPT-4 provides a more sophisticated and comprehensive understanding of customer emotions in complex texts. Integrating GPT-4 into sentiment analysis workflows can significantly enhance the accuracy and richness of insights derived from customer feedback, supporting better business decisions.

Crude oil prices are linked to significant economic activity in all nations across the world, since changes in crude oil prices usually impact the pricing of other commodities and services. As a result, forecasting crude oil prices has become a primary goal for academics and scientists alike. Crude oil has been the most important commodity in the world market and some countries like Nigeria, has it as the main trading commodity to other countries. Crude oil price fluctuations therefore cause problems on global economies and its effects are far reaching leading to either positive or negative economic growth rates. This study present an intelligent system that predicts the price of crude oil. The method used major economic factors that determine the price per barrel as inputs and outputs the price of crude oil. The data for usage came from the West Texas Intermediate (WTI) dataset, which spanned 24 years, and the experimental findings were quite hopeful, demonstrating that support vector machines could be used to forecast crude oil prices with a reasonable level of accuracy. Particle Swarm Optimization (PSO), Support Vector Machine (SVM), and K-Nearest Neighbors were employed in this investigation (KNN) for predicting Crude oil prices and the accuracy of the K-Nearest Neighbours was found to be higher than the Support Vector Machine by 9%.KeywordsCrude oilDatasetMachine learningEconomicsPrice predition

Electrocardiogram analysis using a combination of statistical, geometric, and nonlinear heart rate variability features

Diabetic Retinopathy (DR) is a microangiopathic complication of diabetes mellitus that can cause visual impairment to permanent blindness. Early detection of DR is essential to prevent disease progression, but conventional methods require time, cost, and expertise that are not always available. This study aims to compare the performance of the Random Forest (RF) and Support Vector Machine (SVM) algorithms in DR classification based on texture features extracted from retinal fundus images. The dataset used consists of 3,000 retinal fundus images obtained from the Kaggle platform, divided into 2,400 training data and 600 test data. Image preprocessing includes conversion to grayscale, resizing to a resolution of 128×128 pixels, and normalization. Feature extraction is performed using a combination of Local Binary Pattern (LBP) and Gray Level Co-occurrence Matrix (GLCM) to produce a 14-dimensional feature vector. Performance evaluation uses accuracy, precision, recall, F1-score, ROC curve, and 5-fold cross-validation metrics. The results showed that Random Forest significantly outperformed SVM with an accuracy of 96% compared to 64%, an AUC value of 0.99 compared to 0.72, and an average cross-validation accuracy of 94.5% compared to 63.42%. Random Forest also showed balanced performance in both classes with precision, recall, and F1-score of 0.96, while SVM experienced classification imbalance especially in the disease class. This study proves that Random Forest is a more optimal algorithm for an automatic DR detection system based on fundus image texture features and can support increasing the accessibility of DR screening in areas with limited specialist medical personnel.

In this paper, Maximum Entropy (ME) framework is used to classify text documents. The ME framework has a lot of advantages when compared with other supervised learning algorithms, such as naive Bayes classifier. For example, it makes no inherent conditional independence assumptions between terms. With four labeled data sets, extensive experiments are made to compare the accuracy of ME algorithm with those of naive Bayes and Support Vector Machine (SVM), which are two popular and accurate algorithms in the domain of text classification. The final result is that ME method consistently outperforms naive Bayes and SVM algorithms in accuracy. On the WebKB and Industry Vector data sets, the accuracy of ME algorithm increases from 81.38% to 85.52% and from 85.73% to 89.78% respectively. On the third 20 Newsgroups data set, our experimental result is opposite to that of Nigam et al. For the last Reuters-21578 data set, the accuracy of ME algorithm increases from 94.76% to 96.16%.