Social graph diffusion and disentangled representations learning for multimodal recommendation

In social networks, users can easily share information and express their opinions. Given the huge amount of data posted by many users, it is difficult to search for relevant information. In addition to individual posts, it would be useful if we can recommend groups of people with similar interests. Past studies on user preference learning focused on single-modal features such as review contents or demographic information of users. However, such information is usually not easy to obtain in most social media without explicit user feedback. In this paper, we propose a multimodal feature fusion approach to implicit user preference prediction which combines text and image features from user posts for recommending similar users in social media. First, we use the convolutional neural network (CNN) and TextCNN models to extract image and text features, respectively. Then, these features are combined using early and late fusion methods as a representation of user preferences. Lastly, a list of users with the most similar preferences are recommended. The experimental results on real-world Instagram data show that the best performance can be achieved when we apply late fusion of individual classification results for images and texts, with the best average top-k accuracy of 0.491. This validates the effectiveness of utilizing deep learning methods for fusing multimodal features to represent social user preferences. Further investigation is needed to verify the performance in different types of social media.

Multi-modal recommendation aims at leveraging data of auxiliary modalities (e.g., linguistic descriptions and images) to enhance the representations of items, thereby accurately recommending items that users prefer from the vast expanse of Web-based data. Current multi-modal recommendation methods typically utilize multi-modal features to assist in learning item representations in a direct manner. However, the superfluous semantics in multi-modal features are ignored, resulting in the inclusion of excessive redundancy within the representations of items. Moreover, we disclose that multi-modal features of items rarely contain user-item interaction information. Hence, during the interaction among different item features, the user-item interaction information in ID-based representations diminishes, leading to the degeneration of recommendation performance. To this end, we propose a novel multi-modal recommendation approach, which compresses representations of extra modalities under the guidance of solid theoretical analysis and leverages two auxiliary multi-modal graphs to integrate user-item interaction information into multi-modal features. Empirical experiments on three multi-modal recommendation datasets demonstrate that our method outperforms benchmarks.

LLMs for vision and NLP domain has been popular by the widespread use of ChatGPT and GPT-4. This paper tackles to build LLMs for sensor domain of one-dimensional signals whose downstream task is activity recognition and emotion detection. We propose a new architecture of Transformer-based self-supervised learner which we name SENvT. This SENvT builds the LLMs for sensor data using 7 pretext objectives in multi-task learning together with contrastive learning. Experimental results show these three. First, we obtained better results for contrastive learning and the masked token task but not for other pretext tasks. Second, the masked token task was better in 60% rather than in 10%. Third, the RGW worked best in accuracy while the masked token task worked best in F1.

Online users are typically active on multiple social media networks (SMNs), which constitute a multiplex social network. With improvements in cybersecurity awareness, users increasingly choose different usernames and provide different profiles on different SMNs. Thus, it is becoming increasingly challenging to determine whether given accounts on different SMNs belong to the same user; this can be expressed as an interlayer link prediction problem in a multiplex network. To address the challenge of predicting interlayer links, feature or structure information is leveraged. Existing methods that use network embedding techniques to address this problem focus on learning a mapping function to unify all nodes into a common latent representation space for prediction; positional relationships between unmatched nodes and their common matched neighbors (CMNs) are not utilized. Furthermore, the layers are often modeled as unweighted graphs, ignoring the strengths of the relationships between nodes. To address these limitations, we propose a framework based on multiple types of consistency between embedding vectors (MulCEVs). In MulCEV, the traditional embedding-based method is applied to obtain the degree of consistency between the vectors representing the unmatched nodes, and a proposed distance consistency index based on the positions of nodes in each latent space provides additional clues for prediction. By associating these two types of consistency, the effective information in the latent spaces is fully utilized. In addition, MulCEV models the layers as weighted graphs to obtain representation. In this way, the higher the strength of the relationship between nodes, the more similar their embedding vectors in the latent representation space will be. The results of our experiments on several real-world and synthetic datasets demonstrate that the proposed MulCEV framework markedly outperforms current embedding-based methods, especially when the number of training iterations is small.

In many image-related tasks, learning expressive and discriminative representations of images is essential, and deep learning has been studied for automating the learning of such representations. Some user-centric tasks, such as image recommendations, call for effective representations of not only images but also preferences and intents of users over images. Such representations are termed hybrid and addressed via a deep learning approach in this paper. We design a dual-net deep network, in which the two sub-networks map input images and preferences of users into a same latent semantic space, and then the distances between images and users in the latent space are calculated to make decisions. We further propose a comparative deep learning (CDL) method to train the deep network, using a pair of images compared against one user to learn the pattern of their relative distances. The CDL embraces much more training data than naive deep learning, and thus achieves superior performance than the latter, with no cost of increasing network complexity. Experimental results with real-world data sets for image recommendations have shown the proposed dual-net network and CDL greatly outperform other state-of-the-art image recommendation solutions.

Synthetic aperture radar (SAR) maritime target classification serves as a critical component in modern maritime surveillance. While deep learning networks, particularly convolutional neural networks (CNNs), have driven substantial progress in this domain, three key challenges constrain their performance and practical deployment: 1) In SAR maritime images, complex inshore backgrounds and speckle noise are prevalent. Targets such as ships span a wide range of scales due to different imaging resolutions and intrinsic size variability, exacerbating inter-class similarity and intra-class variability, 2) Labeled data for SAR maritime target classification are scarce, and sensor imaging modes differ markedly across platforms, and 3) Existing CNNs that fuse traditional hand-crafted features often explicitly treat hand-crafted feature extraction as a necessary component of the network and primarily focus on classification performance, overlooking the requirement to efficiently leverage their feature extraction capabilities in downstream tasks. To overcome these challenges, this article proposes a novel SAR maritime target classification network (MBLKNet) based on large kernel convolution and multi-task self-supervised learning. In MBLKNet, four improved designs for network structure are proposed to enhance classification accuracy: 1) macro design, 2) multi-branch large kernel convolution module (MBLKCM), 3) lightweight channel-interactive multi-layer perceptron (LCIMLP), and 4) micro design. In addition, a multi-resolution unlabeled SAR maritime target dataset (SL-SARShip) and a masked image modeling framework, HOGSparK, are proposed to enable the pre-training of MBLKNet under joint supervision of pixel and HOG features. Comparison results on OpenSARShip 2.0 and FUSAR-Ship with state-of-the-art networks, as well as experiments on SSDD for SAR downstream target detection and instance segmentation, demonstrate that the proposed MBLKNet achieves superior performance and strong feature extraction ability.

Many real-world large-scale regression problems can be formulated as Multi-task Learning (MTL) problems with a massive number of tasks, as in retail and transportation domains. However, existing MTL methods still fail to offer both the generalization performance and the scalability for such problems. Scaling up MTL methods to problems with a tremendous number of tasks is a big challenge. Here, we propose a novel algorithm, named Convex Clustering Multi-Task regression Learning (CCMTL), which integrates with convex clustering on the k-nearest neighbor graph of the prediction models. Further, CCMTL efficiently solves the underlying convex problem with a newly proposed optimization method. CCMTL is accurate, efficient to train, and empirically scales linearly in the number of tasks. On both synthetic and real-world datasets, the proposed CCMTL outperforms seven state-of-the-art (SoA) multi-task learning methods in terms of prediction accuracy as well as computational efficiency. On a real-world retail dataset with 23,812 tasks, CCMTL requires only around 30 seconds to train on a single thread, while the SoA methods need up to hours or even days.

Category-based information filtering is ground on the representation of user preferences according to a set of categories of similar items. The use of type 1 fuzzy sets provides a good method to represent categories when only one static interpretation of them is considered. This representation is not enough when documents do not have the same meaning for two different users because there are some degrees of subjectivity. On the other hand, type 2 fuzzy sets have been successfully applied to manage uncertainty more effectively than type-1 fuzzy sets in several environments. This paper presents a method to manage efficiently uncertainties in the filtering process in environments where there is a constant flow of new information (news, e-mail, etc.) and multiple users are involved. The proposed solution is based on the extension of the categories-based filtering method using interval type 2 fuzzy sets for representing each category and the user preferences. Experimental results, that illustrate the feasibility of this approach, are provided.

Fake news that combines text and images has a better story-telling ability than text-only fake news, making it more deceptive and easier to spread maliciously. Therefore, multi-modal fake news detection has become a new hot topic. There are two main challenges in this task. First, the traditional pre-training model BERT has more parameters and a relatively slow training speed, which limits the level of extracted text features. Second, in the multi-modal form, the fusion process is only a simple splicing of visual and textual features of news, and the obtained multi-modal features are insufficient to express the complementarity between multi-modal data and may have redundant information, potentially leading to biased detection results.In order to solve the above issues, we propose ALB-MCF, using ALBERT and Multi-modal Circulant Fusion(MCF) for fake news detection. ALB-MCF consists of four main modules: a multi-modal feature extractor, a multi-modal feature fusion, a fake news detection and a domain classifier. Specifically, the multi-modal feature extractor innovatively uses a pre-trained ALBERT model to extract text features and a pre-trained VGG19 model to extract visual features. Then, the text features and the visual features are fused into a multi-modal feature representation by MCF, which improves the fusion capability while avoiding an increase in parameters and computational cost. Finally, the multi-modal features are fed into the detector to detect fake news. The role of the domain classifier is mainly to remove event-specific features while retaining shared features between events, thus providing effective detection of emerging events. We have conducted extensive experiments on two real-world datasets. The results demonstrated that our model can handle multi-modal data more effectively, thus improving the accuracy of fake news detection.

A multimodal recommendation system (MRS), which leverages rich multimodal information to model user preferences, has recently attracted significant research interest. Most existing MRSs focus primarily on developing sophisticated encoders for feature extraction, typically relying on simple aggregation of interaction-based features for final predictions. However, this conventional paradigm fails to account for the critical semantic difference between high- and low-rating interactions: while high ratings indicate user preference, low ratings explicitly convey dissatisfaction. Such oversight of negative feedback semantics may significantly limit the system’s recommendation performance. Recently, sign graphs—which model positive and negative feedback signals separately—have gained considerable attention. Inspired by this approach, we propose Sign-Aware Multimodal Graph Recommendation (SiMGR), a novel framework incorporating signed graphs into multimodal recommendation systems. SiMGR fuses multimodal features with signed interactions in a unified graph framework by integrating modality-specific representations and applying user-level thresholds to separate positive and negative subgraphs. A balanced pseudo-edge augmentation strategy is introduced to alleviate sparsity and enhance generalization. Experiments on three public multimodal recommendation datasets show that SiMGR outperforms state-of-the-art baselines, achieving an average 4.28% improvement in NDCG@20.

Context-aware Recommender Systems aim to provide users with the most adequate recommendations for their current situation. However, an exact context obtained from a user could be too specific and may not have enough data for accurate rating prediction. This is known as the data sparsity problem. Moreover, often user preference representation depends on the domain or the specific recommendation approach used. Therefore, a big effort is required to change the method used. In this paper we present a new approach for contextual pre-filtering (i.e. using the current context to select a relevant subset of data). Our approach can be used with existing recommendation algorithms. It is based on two ontologies: Recommender System Context ontology, which represents the context, and Contextual Ontological User Profile ontology, which represents user preferences. We evaluated our approach through an offline study which showed that when used with well-known recommendation algorithms it can significantly improve the accuracy of prediction.

The cold start problem in recommender systems is a long-standing challenge, which requires recommending to new users (items) based on attributes without any historical interaction records. In these recommendation systems, warm users (items) have privileged collaborative signals of interaction records compared to cold start users (items), and these Collaborative Filtering (CF) signals are shown to have competing performance for recommendation. Many researchers proposed to learn the correlation between collaborative signal embedding space and the attribute embedding space to improve the cold start recommendation, in which user and item categorical attributes are available in many online platforms. However, the cold start recommendation is still limited by two embedding spaces modeling and simple assumptions of space transformation. As user-item interaction behaviors and user (item) attributes naturally form a heterogeneous graph structure, in this paper, we propose a privileged graph distillation model (PGD). The teacher model is composed of a heterogeneous graph structure for warm users and items with privileged CF links. The student model is composed of an entity-attribute graph without CF links. Specifically, the teacher model can learn better embeddings of each entity by injecting complex higher-order relationships from the constructed heterogeneous graph. The student model can learn the distilled output with privileged CF embeddings from the teacher embeddings. Our proposed model is generally applicable to different cold start scenarios with new user, new item, or new user-new item. Finally, extensive experimental results on the real-world datasets clearly show the effectiveness of our proposed model on different types of cold start problems, with average 6.6%, 5.6%, and 17.1% improvement over state-of-the-art baselines on three datasets, respectively.

Ordinal classification refers to classification problems in which the classes have a natural order imposed on them because of the nature of the concept studied. Some ordinal classification approaches perform a projection from the input space to one-dimensional (latent) space that is partitioned into a sequence of intervals (one for each class). Class identity of a novel input pattern is then decided based on the interval its projection falls into. This projection is trained only indirectly as part of the overall model fitting. As with any other latent model fitting, direct construction hints one may have about the desired form of the latent model can prove very useful for obtaining high-quality models. The key idea of this letter is to construct such a projection model directly, using insights about the class distribution obtained from pairwise distance calculations. The proposed approach is extensively evaluated with 8 nominal and ordinal classifiers methods, 10 real-world ordinal classification data sets, and 4 different performance measures. The new methodology obtained the best results in average ranking when considering three of the performance metrics, although significant differences are found for only some of the methods. Also, after observing other methods of internal behavior in the latent space, we conclude that the internal projections do not fully reflect the intraclass behavior of the patterns. Our method is intrinsically simple, intuitive, and easily understandable, yet highly competitive with state-of-the-art approaches to ordinal classification.

Machine-Learning-as-a-Service has become increasingly popular, with Recommendation-as-a-Service as one of the representative examples. In such services, providing privacy protection for the users is an important topic. Reviewing privacy-preserving solutions which were proposed in the past decade, privacy and machine learning are often seen as two competing goals at stake. Though improving cryptographic primitives (e.g., secure multi-party computation (SMC) or homomorphic encryption (HE)) or devising sophisticated secure protocols has made a remarkable achievement, but in conjunction with state-of-the-art recommender systems often yields far-from-practical solutions. We tackle this problem from the direction of machine learning. We aim to design crypto-friendly recommendation algorithms, thus to obtain efficient solutions by directly using existing cryptographic tools. In particular, we propose an HE-friendly recommender system, refer to as CryptoRec, which (1) decouples user features from latent feature space, avoiding training the recommendation model on encrypted data; (2) only relies on addition and multiplication operations, making the model straightforwardly compatible with HE schemes. The properties turn recommendation-computations into a simple matrix-multiplication operation. To further improve efficiency, we introduce a sparse-quantization-reuse method which reduces the recommendation-computation time by $9\times$ (compared to using CryptoRec directly), without compromising the accuracy. We demonstrate the efficiency and accuracy of CryptoRec on three real-world datasets. CryptoRec allows a server to estimate a user's preferences on thousands of items within a few seconds on a single PC, with the user's data homomorphically encrypted, while its prediction accuracy is still competitive with state-of-the-art recommender systems computing over clear data. Our solution enables Recommendation-as-a-Service on large datasets in a nearly real-time (seconds) level.

Many real-world datasets are labeled with natural orders, i.e., ordinal labels. Ordinal regression is a method to predict ordinal labels that finds a wide range of applications in data-rich domains, such as natural, health and social sciences. Most existing ordinal regression approaches work well for independent and identically distributed (IID) instances via formulating a single ordinal regression task. However, for heterogeneous non-IID instances with well-defined local geometric structures, e.g., subpopulation groups, multi-task learning (MTL) provides a promising framework to encode task (subgroup) relatedness, bridge data from all tasks, and simultaneously learn multiple related tasks in efforts to improve generalization performance. Even though MTL methods have been extensively studied, there is barely existing work investigating MTL for heterogeneous data with ordinal labels. We tackle this important problem via sparse and deep multi-task approaches. Specifically, we develop a regularized multi-task ordinal regression (MTOR) model for smaller datasets and a deep neural networks based MTOR model for large-scale datasets. We evaluate the performance using three real-world healthcare datasets with applications to multi-stage disease progression diagnosis. Our experiments indicate that the proposed MTOR models markedly improve the prediction performance comparing with single-task ordinal regression models.