DHFM: Diversity Enhanced Hypergraph Factorization Machines for Feature Interaction Modeling

Factorization machine (FM) is a prevalent approach to modelling pairwise (second-order) feature interactions when dealing with high-dimensional sparse data. However, on the one hand, FMs fail to capture higher-order feature interactions suffering from combinatorial expansion. On the other hand, taking into account interactions between every pair of features may introduce noise and degrade the prediction accuracy. To solve these problems, we propose a novel approach, the graph factorization machine (GraphFM), which naturally represents features in the graph structure. In particular, we design a mechanism to select beneficial feature interactions and formulate them as edges between features. Then the proposed model, which integrates the interaction function of the FM into the feature aggregation strategy of the graph neural network (GNN), can model arbitrary-order feature interactions on graph-structured features by stacking layers. Experimental results on several real-world datasets demonstrate the rationality and effectiveness of our proposed approach. The code and data are available at https://github.com/CRIPAC-DIG/GraphCTR.

Click-through rate (CTR) prediction is crucial for computing advertisement and recommender systems. The key challenge of CTR prediction is to accurately capture user interests and deliver suitable advertisements to the right people. However, there are an immense number of features in CTR prediction datasets, which hardly fit when only using an individual feature. To solve this problem, feature interaction that combines several features via an operation is introduced to enhance prediction performance. Many factorizations machine-based models and deep learning methods have been proposed to capture feature interaction for CTR prediction. They follow an enumeration-filter pattern that could not determine the appropriate order of feature interaction and useful feature interaction. The attention logarithmic network (ALN) is presented in this paper, which uses logarithmic neural networks (LNN) to model feature interactions, and the squeeze excitation (SE) mechanism to adaptively model the importance of higher-order feature interactions. At first, the embedding vector of the input was absolutized and a very small positive number was added to the zeros of the embedding vector, which made the LNN input positive. Then, the adaptive-order feature interactions were learned by logarithmic transformation and exponential transformation in the LNN. Finally, SE was applied to model the importance of high-order feature interactions adaptively for enhancing CTR performance. Based on this, the attention logarithmic interaction network (ALIN) was proposed for the effectiveness and accuracy of CTR, which integrated Newton’s identity into ALN. ALIN supplements the loss of information, which is caused by the operation becoming positive and by adding a small positive value to the embedding vector. Experiments are conducted on two datasets, and the results prove that ALIN is efficient and effective.

Feature interaction tends to have a wide range of consequences and effects on a feature model and its applications. While these may often be intended, it is also true that feature validity can be violated, one way or another, by feature interactions (Shah & Mäntylä, 1995, Gao & Shah, 1998, Lee & Kim, 1998). They may affect the semantics of a feature, ranging from slight changes in actual parameter values, to some substantial alterations to both geometry and topology or even complete suppression of its contribution to the model shape. To certain extent, successful applications of feature recognition and feature-based techniques have been hindered by interactions among the features. Feature interaction was first studied in relation to feature recognition systems. As an alternative to feature recognition, feature-based design methodology has also become prevalent in recent years. Although a number of successful and commercially available feature-based design systems have been reported, current CAD technology is still unable to provide an effective solution for fully handling the complexity of feature interactions. Very often in a feature-based design system, the interaction between two features gives rise to an unintended feature, nullifying the one-to-one mapping from design features to manufacturing features. The resulting manufacturing feature is usually of a form that the system cannot handle or represent. Thus feature interaction resolution is equally essential for a feature-based design system (Dereli & Baykasoglu, 2004). As discussed in Chapter IV, features can be represented either as a set of faces or as a volume. The interactions between surface features are different from those occurring between volumetric features. This chapter discusses different types of interactions that arise from these two feature representation schemes and uses the interacting entities to classify them. There are two types of surface feature interactions, basic feature interaction and complex feature interaction. Three types of basic feature interactions are discussed. They are nested, overlapping, and intersecting types. Interacting patches are used to classify volumetric feature interactions. These interacting patches can be of a containing, contained, or overlapping type. The significance of feature interactions lies in their effect on the machining sequence of the features involved. This is also discussed in this chapter. When features are close to each other but do not share any geometric entities, interactions may also happen for structural reasons. This type of feature interaction can be called interaction by vicinity. The main aim of this chapter is to take a holistic approach toward feature interaction solutions. The example parts used are from the “Catalogue of the NIST (National Institute of Standards and Technology) Design, Planning and Assembly Repository” (Regli & Gaines, 1996). A case study is provided in the end of the chapter.

Recently, Graph Neural Networks (GNNs) have been widely used for fraud detection. GNNs first generate node embedding by aggregating neighboring information under different relations, and then use the final node embedding to detect the node’s suspiciousness. However, traditional GNNs employing only a single type of aggregator fail to capture neighbor information from multiple perspectives and treating different relations equally inevitably weakens the semantic information of heterogeneous graphs. Meanwhile, expressive ability of GNNs is limited by using conventional concatenating or averaging operations to update the center node. Also, camouflaged entities could damage GNN-based models. To handle these problems, a novel heterogeneous GNN model called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multiple Aggregators and Feature Interactions Network</i> (MAFI) is proposed in this paper to conduct fraud detection tasks. Concretely, multiple types of aggregators are applied on different relations to aggregate neighbor information and aggregator-level attention is utilized to learn the importance of different aggregators. Also, relation-level attention is leveraged to learn the importance of each relation. Besides, conventional update operations are replaced with vector-wise implicit and explicit feature interactions. Moreover, a trainable neighbor sampler is employed to filter camouflaged fraudsters. Comprehensive experiments on two real-world fraud datasets indicate that the proposed MAFI outperforms existing GNN-based fraud detectors.

Modeling high-order feature interactions is crucial for click-through rate (CTR) prediction, yet traditional approaches typically predefine a maximum interaction order and exhaustively enumerate feature combinations up to that order. This paradigm depends heavily on prior domain knowledge to delimit the interaction space and incurs substantial computational overhead. As a result, conventional CTR models face a persistent tension between enriching representations with complex high-order interactions and keeping computation tractable. To address this dual challenge, this study introduces the Kolmogorov-Arnold Represented Sparse Efficient Interaction Network (KarSein). Drawing inspiration from the learnable activation mechanism in the Kolmogorov-Arnold Network (KAN), KarSein leverages this mechanism to adaptively transform low-order basic features into high-order feature interactions, offering a novel approach to feature interaction modeling. KarSein extends the capabilities of KAN by introducing a more efficient architecture that significantly reduces computational costs while accommodating two-dimensional embedding vectors as feature inputs. Furthermore, it overcomes the limitation of KAN’s its inability to spontaneously capture multiplicative relationships among features. Extensive experiments highlight the superiority of KarSein, demonstrating its ability to surpass not only the vanilla implementation of KAN in CTR prediction tasks but also other baseline methods. Remarkably, KarSein achieves exceptional predictive accuracy while maintaining a highly compact parameter size and minimal computational overhead. Moreover, KarSein retains the key advantages of KAN, such as strong interpretability and structural sparsity. As the first systematic adaptation of KAN to CTR prediction, KarSein offers a practical, parameter-efficient, and interpretable alternative for modeling complex feature interactions in large-scale recommendation systems 1 .

Twisted convolutional networks (TCNs): Enhancing feature interactions for non-spatial data classification.

With the growth of high-dimensional sparse data in web-scale recommender systems, the computational cost to learn high-order feature interaction in CTR prediction task largely increases, which limits the use of high-order interaction models in real industrial applications. Some recent knowledge distillation based methods transfer knowledge from complex teacher models to shallow student models for accelerating the online model inference. However, they suffer from the degradation of model accuracy in knowledge distillation process. It is challenging to balance the efficiency and effectiveness of the shallow student models. To address this problem, we propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. The proposed lightweight student model DAGFM can learn arbitrary explicit feature interactions from teacher networks, which achieves approximately lossless performance and is proved by a dynamic programming algorithm. Besides, an improved general model KD-DAGFM+ is shown to be effective in distilling both explicit and implicit feature interactions from any complex teacher model. Extensive experiments are conducted on four real-world datasets, including a large-scale industrial dataset from WeChat platform with billions of feature dimensions. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments, showing the superiority of DAGFM to deal with the industrial scale data in CTR prediction task. Our implementation code is available at: https://github.com/RUCAIBox/DAGFM.

Extracting more information from feature interactions is essential to improve click-through rate (CTR) prediction accuracy. Although deep learning technology can help capture high-order feature interactions, the combination of features lacks interpretability. In this paper, we propose a multi-semantic feature interaction learning network (MeFiNet), which utilizes convolution operations to map feature interactions to multi-semantic spaces to improve their expressive ability and uses an improved Squeeze &amp; Excitation method based on SENet to learn the importance of these interactions in different semantic spaces. The Squeeze operation helps to obtain the global importance distribution of semantic spaces, and the Excitation operation helps to dynamically re-assign the weights of semantic features so that both semantic diversity and feature diversity are considered in the model. The generated multi-semantic feature interactions are concatenated with the original feature embeddings and input into a deep learning network. Experiments on three public datasets demonstrate the effectiveness of the proposed model. Compared with state-of-the-art methods, the model achieves excellent performance (+0.18% in AUC and -0.34% in LogLoss VS DeepFM; +0.19% in AUC and -0.33% in LogLoss VS FiBiNet).

Modern automotive software systems are large, complex, and feature rich; they can contain over 100 million lines of code, comprising hundreds of features distributed across multiple electronic control units (ECUs), all operating in parallel and communicating over a CAN bus. Because they are safety-critical systems, the problem of possible Feature Interactions (FIs) must be addressed seriously; however, traditional detection approaches using dynamic analyses are unlikely to scale to the size of these systems. We are investigating an approach that detects static source-code patterns that are symptomatic of FIs. The tools report Feature-Interaction warnings, which can be investigated further by engineers to determine if they represent true FIs and if those FIs are problematic.In this paper, we present our preliminary toolchain for FI detection. First, we extract a collection of static “facts” from the source code, such as function calls, variable assignments, and messages between features. Next, we perform relational algebra transformations on this factbase to infer additional “facts” that represent more complicated design information about the code, such as potential information flows and data dependencies; then, the full collection of “facts” is matched against a curated set of patterns for FI symptoms. We present a set of five patterns for FIs in automotive software as well a case study in which we applied our tools to the Autonomoose autonomous-driving software, developed at the University of Waterloo. Our approach identified 1,444 possible FIs in this codebase, of which 10% were classified as being probable interactions worthy of further investigation.

A unified dual-view knowledge-guided sentiment interaction networks for aspect-based sentiment analysis.

In the field of recommendation systems, click-through rate (CTR) prediction is essential for measuring user engagement and product interest. However, input features are typically high-dimensional and sparse, requiring effective modeling of high-order feature interactions. Existing methods learn low-dimensional representations and identify useful feature combinations, but they often struggle to capture implicit interactions in non-Euclidean spaces, leading to noisy feature relationships, reduced interpretability, and suboptimal prediction performance. To address these challenges, we introduce a similarity-based adaptive graph attention network (SGAT) for CTR prediction. The SGAT employs a dual-attention mechanism that integrates similarity-based adaptive attention with softmax attention to selectively emphasize beneficial feature interactions while suppressing irrelevant ones. This strategy reduces noise, enhances interaction modelling, and mitigates overfitting. Experimental results on two public benchmark datasets show that SGAT achieves superior performance compared with several state-of-the-art baseline methods.

In recommendation systems, it is of vital importance to comprehensively consider various aspects of information to make accurate recommendations for users. When the low-order feature interactions between items are insufficient, it is necessary to mine information to learn higher-order feature interactions. In addition, to distinguish the different importance levels of feature interactions, larger weights should be assigned to features with larger contributions to predictions, and smaller weights to those with smaller contributions. Therefore, this paper proposes a neural attention model for recommendation (NAM), which deepens factorization machines (FMs) by adding an attention mechanism and fully connected layers. Through the attention mechanism, NAM can learn the different importance levels of low-order feature interactions. By adding fully connected layers on top of the attention component, NAM can model high-order feature interactions in a nonlinear way. Experiments on two real-world datasets demonstrate that NAM has excellent performance and is superior to FM and other state-of-the-art models. The results demonstrate the effectiveness of the proposed model and the potential of using neural networks for prediction under sparse data.

In recent years, leveraging the characteristics of users’ historical behavior to predict click-through rates (CTRs) has become a key point of interest in studies of recommender systems. Although theoretical and experimental investigations of CTR models have increased substantially, most models focus on linear feature interaction; however, crucial user characteristics in the real world are discovered implicitly by non-linear features. In this paper, we propose a novel model that integrates the advantages of linear and non-linear feature interaction. Our deep factorization machines network with non-linear interaction for recommend systems (DFNR) model identifies non-linear feature interactions by designing a new Non-linear interaction (NL-interaction) layer. We also incorporate a deeper multilayer perceptron (MLP) than other CTR models, which yields more accurate information about higher-order feature interactions. The MLP in the proposed model is unique because we use the residual structure to correct problems caused by a deeper network structure. Findings show that our DFNR model performs better on a CTR prediction task compared to other models. Results demonstrate the effective-ness of our model based on its non-linear interaction layer and deeper neural network architecture.

Previous studies have shown that optimized tool paths based on behavior of individual features and feature interactions can be used to improve the accuracy of features in parts produced by single point incremental forming. These tool paths are generated with compensated CAD files of the part, which result from a prediction of deviations of individual features. However, in order to improve the accuracy of an entire part, it is important to systematically look at behavior of all the individual features and all feasible interactions between features. In this paper, the authors present a graph topology approach to integrating the effects of the behavior of all features present in a part. For any given part, a conceptual graph is constructed representing all the features and connecting them based on their spatial locations with conceptual relations. Next, all possible feature interactions based on the generated graph are analyzed, and the deviations due to the feasible interactions in an uncompensated test are predicted. Depending on the feature types and interactions present, a comprehensive strategy for accurate part manufacture is generated. This strategy may be composed of a selection of one or more complementary tool path strategies for compensating the anticipated deviations on the part. Case studies illustrating improvement in accuracy of parts produced by this technique are discussed next to justify the use of the graph based approach.

As a software product line evolves, it typically introduces new features and includes new products over time. A known cause of software aging in product lines is the introduction of new features that interact in unplanned and even risky ways with the existing features. This can lead to failures, performance degradation, and hazardous states in a new product. Software product line developers currently identify new, unwanted feature interactions primarily in the testing of each new product. This incurs significant costs, comes late in development, and does not exploit the knowledge of prior feature interactions within a product line. The contribution of our paper is to leverage knowledge of prior feature interactions in a product line, together with similarity measures between the features in known feature interactions and the new features, in order to detect similar feature interactions in a new product much earlier in the development process. Results from application to a case study from the literature show that this approach accurately detected 73% of feature interactions. This small study suggests that using similarity measures at the feature level within a product line to detect problematic interactions involving a new feature can effectively reduce this cause of aging in a software product line.