Chapter 6 - Feature engineering

Recent technological advances allowed an exponential expansion of biological sequence data, and the extraction of meaningful information through Machine Learning (ML) algorithms. This knowledge improved the understanding of the mechanisms related to several fatal diseases, e.g., Cancer and COVID-19, helping to develop innovative solutions, such as CRISPR-based gene editing, coronavirus vaccine, and precision medicine. These advances benefit our society and economy, directly impacting peoples lives in various areas, such as health care, drug discovery, forensic analysis, and food analysis. Nevertheless, ML approaches applied to biological data require representative, quantitative, and informative features. Necessarily, as many ML algorithms can handle only numerical data, sequences need to be translated into a feature vector. This process, known as feature extraction, is a fundamental step for the elaboration of high-quality ML-based models in bioinformatics, by allowing the feature engineering stage, with the design and selection of suitable features. Feature engineering, ML algorithm selection, and hyperparameter tuning are often manual and time-consuming processes, requiring extensive domain knowledge, and performed manually by a human expert. To deal with this problem, we developed a new package, BioAutoML, which automatically runs an end-to-end ML pipeline. BioAutoML extracts numerical and informative features from biological sequence databases, automating feature selection, recommendation of ML algorithm(s), and tuning of hyperparameters, using Automated ML (AutoML). BioAutoML has two components, divided into four modules, (1) automated feature engineering (feature extraction and selection modules) and (2) Metalearning (algorithm recommendation and hyperparameter tuning modules). Our experimental results, assessing the relevance of our proposal, indicate robust results for different problem domains, such as SARS-CoV-2, anticancer peptides, HIV sequences, and non-coding RNAs. According to our systematic review, our proposal is innovative compared to available studies in the literature, being the first study to propose automated feature engineering and metalearning for biological sequences. BioAutoML has a high potential to significantly reduce the expertise required to use ML pipelines, aiding researchers in combating diseases, particularly in low- and middle-income countries. This initiative can provide biologists, physicians, epidemiologists, and other stakeholders with an opportunity for widespread use of these techniques to enhance the health and well-being of their communities.

Detecting loan fraud is a subject frequently raised in scientific research, and to which financial institutions attach great importance, given the losses they avoid by applying an efficient fraud detection. The purpose of this paper is to compare a machine learning algorithm with statistical models in fraud detection and describe a machine learning approach to loan fraud detection using data mining techniques. The paper proposes an approach using data mining techniques for machine learning in order to improve performance. In the training stage, the authors used two datasets. The first one is a raw data, and the second is a pre-processed data with feature engineering and selection methods. The results show a significant improvement in performance of all classifiers once tested on pre-processed data. Finally, they conclude that applying machine learning algorithms directly to raw data gives bad results for statistical model comparing to machine learning algorithms, and the proposed approach using feature engineering and selection techniques contributes to improve performance.

In the practical implementation of control of electromyography (sEMG) driven devices, algorithms should recognize the human’s motion from sEMG with fast speed and high accuracy. This study proposes two feature engineering (FE) techniques, namely, feature-vector resampling and time-lag techniques, to improve the accuracy and speed of least square support vector machine (LSSVM) for wrist palmar angle estimation from sEMG feature. The root mean square error and correlation coefficients of LSSVM with FE are 9.50 2.32 degree and 0.971 0.018 respectively. The average training time and average execution time of LSSVM with FE in processing 12600 sEMG points are 0.016 s and 0.053 s respectively. To evaluate the proposed algorithm, its estimation results are compared with those of three other methods, namely, LSSVM, radial basis function (RBF) neural network, and RBF with FE. Experimental results verify that introduction of time-lag into feature vector can greatly improve the estimation accuracy of both RBF and LSSVM; meanwhile the application of feature-vector resampling technique can significantly increase the training and execution speed of RBF neural network and LSSVM. Among different algorithms applied in this study, LSSVM with FE techniques performed best in terms of training and execution speed, as well as estimation accuracy.

Feature engineering and selection is a critical step in the implementation of any machine learning system. In application areas such as intrusion detection for cybersecurity, this task is made more complicated by the diverse data types and ranges presented in both raw data packets and derived data fields. Additionally, the time and context specific nature of the data requires domain expertise to properly engineer the features while minimizing any potential information loss. Many previous efforts in this area naively apply techniques for feature engineering that are successful in image recognition applications. In this work, we use network packet dataflows from the Defense Research and Engineering Network (DREN) and the Engineer Research and Development Center’s (ERDC) high performance computing systems to experimentally analyze various methods of feature engineering. The results of this research provide insight on the suitability of the features for machine learning based cybersecurity applications.

Feature engineering involves extracting information from raw-data to use in machine learning or deep learning algorithms through feature transformation, feature generation or feature extraction, feature construction, feature selection, etc. Feature engineering optimizes the feature space dimensions, thereby reducing complexity. Hence, makes the data suitable for machine learning or deep learning applications: process prediction, detection accuracy, estimation accuracy, and quality of clustering and classi cation. Traditionally machine learning algorithms have been used in various wireless communication appli cations viz spectrum access and sharing, resource allocation, spectrum coverage, capacity optimization, signal intelligence, etc. This chapter presents the need for feature engineering and its application to machine learning and deep learning in wireless communication. The chapter also gives the details of the feature engineering assisted machine-learning algorithm for automatic modulation classi cation and path-loss prediction in wireless communication.KeywordsDeep learningFeature engineeringMachine learningWireless communication

Evaluation of multispectral imaging for freeze damage assessment in strawberries using AI-based computer vision technology

Principles, research status, and prospects of feature engineering for data-driven building energy prediction: A comprehensive review

Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection

The ability of music to spread joy and excitement across lives, makes it widely acknowledged as the human race's universal language. The phrase "music genre" is frequently used to group several musical styles together as following a shared custom or set of guidelines. According to their unique preferences, people now make playlists based on particular musical genres. Due to the determination and extraction of appropriate audio elements, music genre identification is regarded as a challenging task. Music information retrieval, which extracts meaningful information from music, is one of several real - world applications of machine learning. The objective of this paper is to efficiently categorise songs into various genres based on their attributes using various machine learning approaches. To enhance the outcomes, appropriate feature engineering and data pre-processing techniques have been performed. Finally, using suitable performance assessment measures, the output from each model has been compared. Compared to other machine learning algorithms, Random Forest along with efficient feature selection and hyperparameter tuning has produced better results in classifying music genres.

In light of the escalating ubiquity of the Internet, the proliferation of cyber-attacks, coupled with their intricate and surreptitious nature, has significantly imperiled network security. Traditional machine learning methodologies inherently exhibit constraints in effectively detecting and classifying multifarious cyber threats. Specifically, the surge in high-dimensional network traffic data and the imbalanced distribution of classes exacerbate the predicament of ideal classification performance. Notably, the presence of redundant information within network traffic data undermines the accuracy of classifiers. To address these challenges, this study introduces a novel approach for intrusion detection classification which integrates advanced techniques of feature engineering and model optimization. The method employs a feature engineering approach that leverages mutual information maximum correlation minimum redundancy (mRMR) feature selection and synthetic minority class oversampling technique (SMOTE) to process network data. This transformation of raw data into more meaningful features effectively addresses the complexity and diversity inherent in network data, enhancing classifier accuracy by reducing feature redundancy and mitigating issues related to class imbalance and the detection of rare attacks. Furthermore, to optimize classifier performance, the paper applies the Optuna method to fine-tune the hyperparameters of the Catboost classifier, thereby determining the optimal model configuration. The study conducts binary and multi-classification experiments using publicly available datasets, including NSL_KDD, UNSW-NB15, and CICIDS-2017. Experimental results demonstrate that the proposed method outperforms traditional approaches regarding accuracy, recall, precision, and F-value. These findings highlight the method’s potential and performance in network intrusion detection.

In the era of digital information, online platforms play a crucial role in shaping public opinion. However, the extensive spread of misinformation and fake news poses a significant challenge, largely fueled by non-credible users. Detecting user credibility is vital for ensuring the reliability of information on these platforms. This study employs supervised machine learning algorithms, leveraging key user features to enhance credibility detection. Feature selection methods, specifically SelectKBest and correlation-based algorithms, are explored for their impact on X-Platform user credibility detection. Utilizing various classifiers, including support vector machine, logistic regression, and XGBoost, experiments are conducted on the ArPFN dataset, which is a labeled, balanced, publicly available dataset. The evaluation includes measures like accuracy, precision, recall, and F1-score to assess efficiency. This research considers feature categories and selection methods with SML to detect their impact on the accuracy of X-Platform user credibility detection, making this research a reference for researchers and practitioners working in the field of SML, feature engineering, and social media analysis. We aim to advance the field’s understanding of effective strategies for mitigating the spread of fake news. The novelty of this study lies in the comprehensive exploration of feature selection methods and their influence on credibility detection, contributing valuable insights for future research in this domain.

The time resolution and prediction accuracy of the power generated by building-integrated photovoltaics are important for managing electricity demand and formulating a strategy to trade power with the grid. This study presents a novel approach to improve short-term hourly photovoltaic power output predictions using feature engineering and machine learning. Feature selection measured the importance score of input features by using a model-based variable importance. It verified that the normative sky index in the weather forecasted data had the least importance as a predictor for hourly prediction of photovoltaic power output. Six different machine-learning algorithms were assessed to select an appropriate model for the hourly power output prediction with onsite weather forecast data. The recurrent neural network outperformed five other models, including artificial neural networks, support vector machines, classification and regression trees, chi-square automatic interaction detection, and random forests, in terms of its ability to predict photovoltaic power output at an hourly and daily resolution for 64 tested days. Feature engineering was then used to apply dropout observation to the normative sky index from the training and prediction process, which improved the hourly prediction performance. In particular, the prediction accuracy for overcast days improved by 20% compared to the original weather dataset used without dropout observation. The results show that feature engineering effectively improves the short-term predictions of photovoltaic power output in buildings with a simple weather forecasting service.

This paper applies an organized flow of feature engineering and machine learning to detect distributed denial-of-service (DDoS) attacks. Feature engineering has a focus to obtain the datasets of different dimensions with significant features, using feature selection methods of backward elimination, chi2, and information gain scores. Different supervised machine learning models are applied on the feature-engineered datasets to demonstrate the adaptability of datasets for machine learning under optimal tuning of parameters within given sets of values. The results show that substantial feature reduction is possible to make DDoS detection faster and optimized with minimal performance hit. The paper proposes a strategic-level framework which incorporates the necessary elements of feature engineering and machine learning with a defined flow of experimentation. The models are also validated with cross-validation and evaluated for area-under-curve analyses. It provides comprehensive solutions which can be trusted to avoid the overfitting and collinearity problems of data while detecting DDoS attacks. In the case study of DDoS datasets, K-nearest neighbors algorithm overall exhibits the best performance followed by support vector machine, whereas low-dimensional datasets of discrete feature types perform better under the Random Forest model as compared to high dimensions with numerical features. The accuracy scores of dataset with the lowest number of features remain competitive with other datasets under all machine learning models, leading to a substantially reduced processing overhead. The experiments show that approximately 68% reduction in the feature space is possible with an impact of only about 0.03% on accuracy.

Diabetes, a chronic illness, has seen an increase in prevalence over the years, posing several health challenges. This study aims to predict diabetes onset using the Pima Indians Diabetes dataset. We implemented several machine learning algorithms, namely Random Forest, Gradient Boosting, XGBoost, LightGBM, and CatBoost. To enhance model performance, we applied a variety of feature engineering techniques, including SelectKBest, Recursive Feature Elimination (RFE), Recursive Feature Elimination with Cross-Validation (RFECV), Forward Feature Selection, and Backward Feature Elimination. RFECV proved to be the most effective method, leading to the selection of the best feature set. In addition, hyperparameter tuning techniques are used to determine the optimal parameters for the models created. Upon training these models with the optimized parameters, XGBoost outperformed the others with an accuracy of 94%, while Random Forest and CatBoost both achieved 92.5%. These results highlight XGBoost's superior predictive power and the significance of thorough feature engineering and model tuning in diabetes prediction.

• Novel FlexiCombFE framework for brain tumor detection using MRI images. • Combines flexible patch-based feature extraction with multiple selectors. • Achieves 99.35% accuracy, outperforming several deep learning approaches. • Modular design allows detailed analysis of classification effects. • Provides interpretable feature engineering alternative to deep learning. Deep learning models are prevalently employed for image classification, but significant opportunities remain within the domain of feature engineering. This study introduces FlexiCombFE, a novel, flexible, combination-based feature engineering framework for brain tumor detection using various fixed-size patch divisions. This framework employs three distinct feature extractors (Local Phase Quantization, Local Binary Pattern, and Pyramidal Histogram of Oriented Gradients) to generate a total of seven primary feature vectors. By using four types of fixed-size patch divisions, 28 feature vectors are generated. Then, three feature selectors (Chi-squared, Neighborhood Component Analysis, and ReliefF) create 84 selected feature vectors. In the classification phase K-nearest neighbors and support vector machine classifiers yield 168 classifier-specific outcomes. An information fusion generates 166-voted outcomes, with the most accurate classification outcome selected as the final output. This self-organizing feature engineering model achieved a classification accuracy of 99.35% on a brain tumor image dataset, outperforming several deep learning approaches. The modular design of the proposed framework allows for detailed analysis of the classification effects of individual methods, providing optimal feature engineering strategies for medical image classification tasks.