The proliferation of artificial intelligence has enabled a diversity of applications that bridge the gap between digital and physical worlds. As physical environments are too complex to model through a single information acquisition approach, it is crucial to fuse multimodal data generated by different sources, such as sensors, devices, systems, and people, to solve a problem in the real world. Unfortunately, it is neither applicable nor sustainable to deploy new resources to collect original data from scratch for every problem. Thus, when data is inadequate in the domain of problem, it is vital to fuse knowledge from multimodal data that is already available in other domains. We call this cross-domain knowledge fusion. Existing research focus on fusing multimodal data in a single domain, supposing the knowledge from different datasets is intrinsically aligned; however, this assumption may not hold in the scenarios of cross-domain knowledge fusion. In this paper, we formally define the cross-domain multimodal data fusion problem, discussing its unique challenges, differences and advantages beyond data fusion in a single domain. We propose a four-layer framework, consisting of Domains, Links, Models and Data layers, answering three key questions: “what to fuse”, “why can be fused”, and “how to fuse”. The Domains Layer selects relevant data from different domains for a given problem. The Links Layer reveals the philosophy of knowledge alignment beyond specific model structures. The Models Layer provides two knowledge fusion paradigms based on the fundamental mechanisms for processing data. The Data Layer turns data of different structures, resolutions, scales and distributions into a consistent representation that can be fed into an AI model. With this framework, we can design solutions that fuse cross-domain multimodal data effectively for solving real-world problems.

To tackle the issue of excessive parameter volumes during fine-tuning of large-scale pre-trained models with full parameters, Parameter-Efficient Fine-Tuning (PEFT) methods have been introduced. The core concept involves freezing the backbone network of the model and updating only a small subset of parameters. This strategy not only decreases the number of parameters needed for training but also delivers performance comparable to Full-Tuning, even surpassing it on certain datasets. However, most popular PEFT methods introduce extra parameters or modules for fine-tuning, which come with inherent limitations. In response, we propose a straightforward and efficient PEFT method called You Can Only Tune Normalization (YONO). YONO focuses solely on tuning the normalization layer and the final classification layer of the model. This method avoids adding extra modules, making it easily applicable to any model without causing inference delays. We extensively tested YONO on 28 benchmark datasets, and the results indicate that it requires significantly fewer parameters compared to other advanced PEFT methods. Additionally, we validated YONO's efficiency and generalizability across various vision models. Finally, we further explore the essence of PEFT methods, whether they learn new knowledge or expose the capabilities that a model has already learned. Our findings suggest that YONO is more sensitive to improvements in dataset quality, making it a promising candidate for future scaling to larger models.

Urban Foundation Models (UFMs) are emerging as a transformative paradigm for understanding and managing complex urban systems, offering new capabilities to improve and transform urban environments. Trajectory data (e.g., vehicle movements) contains rich spatio-temporal patterns of human mobility and urban dynamics that are crucial for understanding city-wide movement behaviors, traffic patterns, and transportation system operations. Effectively modeling such trajectory data is crucial for UFMs, yet this poses significant challenges due to the high-dimensional nature of raw trajectories and their complex spatio-temporal dynamics. Trajectory representation learning addresses these challenges by transforming raw trajectory data into compact, meaningful representations that capture essential mobility patterns and facilitates the application to various urban tasks, such as trajectory similarity computation, travel time estimation, or destination prediction. This is achieved by learning low-dimensional representations from high-dimensional and raw trajectory data. However, existing methods for trajectory representation learning either rely on grid-based or road-based representations, which are inherently different and thus, could lose information contained in the other modality. Moreover, these methods overlook the dynamic nature of urban traffic, relying on static road network features rather than time-varying traffic patterns. In this paper, we propose TIGR , a novel model designed to integrate grid and road network modalities while incorporating spatio-temporal dynamics to learn rich, general-purpose representations of trajectories. We evaluate TIGR on three real-world datasets and demonstrate the effectiveness of combining both modalities by substantially outperforming state-of-the-art methods, i.e., up to 59.34% for trajectory similarity, up to 16.65% for travel time estimation, and up to 14.81% for destination prediction.

In the past few years, infectious diseases like COVID-19 have caused serious distress to the global society and the economy. To prevent its spread, the early detection and assessment of infectious diseases based on molecular tests or antigen testing of bodily have led to countless labor and material costs. Fortunately, with the rapid development of mobile localization and web techniques, the collected massive mobile trajectory data provides a promising solution for detecting positive cases. However, existing mobility data-driven infection case detection methods are limited in terms of modeling the complicated epidemic spreading processes and preserving user privacy of the mobility data. In this paper, we propose a novel graph convolutional networks (GCN) model for detecting high risk infection cases, where we incorporate a spatio-temporal hypergraph to model the complex interaction of individuals. Then, we elaborately design a privacy-preserving framework tightly coupled with the structure of the spatio-temporal hypergraph, which includes a mobility data obfuscation module to protect privacy and an accompanying confidence-aware mechanism to mitigate the consequent performance decline. Moreover, we introduce a causal propagation mechanism to further guarantee the temporal dependency and causal effect of the feature propagation in our spatio-temporal hypergraph, which introduces both the causal transform of node features and the causal gathering of edge features. Finally, extensive experiments on a large mobility dataset collected from location-based services (LBS) show that the proposed model improves the performance of infection case detection by at least \(12.47\%\) when compared with several widely adopted baselines. Besides, our code and datasets are available at the link 1 .

Generating temporally smooth and high-resolution videos is a crucial objective in talking face generation tasks. Diffusion-based generative models have emerged as a prime choice for these tasks due to their ability to produce high-quality outputs. To mitigate the impact of stochasticity in the diffusion process, recent research has predominantly utilized self-attention layers to extract temporal features, ensuring temporal consistency in the generated videos. However, self-attention mechanisms have computational complexity that scales quadratically with video length, leading to high computational costs. This limitation poses significant challenges when attempting to generate longer video sequences using diffusion models. To address this challenge, we propose Talking-DiSSM, an end-to-end method for generating audio-driven talking face videos using State-Space Models (SSMs). This novel framework for conditional video diffusion modeling integrates Bidirectional State-Space Models (Bi-SSM) as temporal modeling modules with linear complexity, effectively capturing complex sequential temporal information and intra-batch sequential interdependencies in videos. Additionally, we employ a simple yet effective batch-overlapped sampling strategy to process input video clips, constructing inter-batch correlations while incorporating reference face clips and landmarks as conditions to ensure stability in the generation process. Extensive experiments demonstrate that Talking-DiSSM generates temporally consistent, high-quality, and identity-preserving talking face videos synchronized with the driving audio, achieving state-of-the-art results compared to existing models.

Data sparsity is a persistent challenge in recommender systems, specially in specific domains like Point-of-Interest (POI) recommendation, where it significantly impacts model performance. While classical recommender systems have used various imputation and data augmentation mechanisms to address data sparsity, these methods have not been extensively explored in the POI recommendation domain. In this work, we propose a generic imputation framework to study the use of data augmentation techniques to generate synthetic check-ins and analyze their effects on the POI recommendation scenario. Our main goal is to enhance the performance of various traditional recommenders by increasing the training set interactions, considering specific characteristics of the domain, such as geographical information. We apply these techniques in six different cities from a global Foursquare check-in dataset, as well as in two additional cities from the Gowalla dataset, and a separate dataset from Yelp, ensuring a comprehensive evaluation across multiple data sources. Our imputation approach evidences improvements for most models. In several cases, these improvements exceeded 100% for ranking accuracy, measured in terms of nDCG, without considerably compromising novelty or diversity. Data and code is released at https://github.com/pablosanchezp/ImputationForPOIRecsys .

Mining temporal patterns from user behaviors has long been investigated, but most of the existing work centers on single-type user-item interactions, such as purchase or click, which fails to take advantage of the user’s diversified interests revealed by various types of behavior. However, capturing patterns from different behavior sequences and modeling the complex inter-correlation between them are non-trivial tasks, as the high sparsity of type-related interactions, multi-seasonality of individual behaviors and time-variant dependency of multi-type activities make it really challenging. To address these challenges, we propose a novel framework that aims to model the M ulti-seasonal M ulti-behavior Dep endencies (MMDep) both within and across the multi-type behavior sequences. In the proposed model, an item co-occurrence matrix factorization strategy is introduced to alleviate the sparsity issue in type-related behavior sequences. And a temporal dependency module that incorporates multi-scale EMA mechanism is utilized to capture the multi-seasonal dependencies within individual sequences. Moreover, a cross-behavior dependency module is employed to learn the time-variant dependency among different behaviors. Extensive experiments on three real-world datasets demonstrate that the proposed MMDep performs significantly better than the state-of-the-art baselines. And it may provide some new insights and tools on how to leverage multi-behavior data for better temporal recommendation.

Precise estimation of treatment effects is crucial for accurately evaluating the intervention. While deep learning models have exhibited promising performance in learning counterfactual representations for treatment effect estimation (TEE), a major limitation in most of these models is that they often overlook the diversity of treatment effects across potential subgroups that have varying treatment effects and characteristics, treating the entire population as a homogeneous group. This limitation restricts the ability to precisely estimate treatment effects and provide targeted treatment recommendations. In this paper, we propose a novel treatment effect estimation model, named SubgroupTE, which incorporates subgroup identification in TEE. SubgroupTE identifies heterogeneous subgroups with different responses and more precisely estimates treatment effects by considering subgroup-specific treatment effects in the estimation process. In addition, we introduce an expectation–maximization (EM)-based training process that iteratively optimizes estimation and subgrouping networks to improve both estimation and subgroup identification. Comprehensive experiments on the synthetic and semi-synthetic datasets demonstrate the outstanding performance of SubgroupTE compared to the existing works for treatment effect estimation and subgrouping models. Additionally, a real-world study demonstrates the capabilities of SubgroupTE in enhancing targeted treatment recommendations for patients with opioid use disorder (OUD) by incorporating subgroup identification with treatment effect estimation.

Knowledge tracing (KT) is a problem of modeling students’ knowledge states to predict their future performance by observing their historical learning interactions. The collection of educational data presents significant challenges, as students’ limited learning engagement restricts the generation of large-scale interaction data, while stringent privacy regulations further limit the availability of student learning sequences from online platforms. Hence, it is crucial to enhance the capabilities of deep learning based KT (DLKT) models by constructing large-scale datasets through the integration of student interaction data across multiple subjects and sources. The success of ChatGPT demonstrates that the decoder-only Transformer architecture is highly effective in capturing complex information from large-scale sequential data. Against this background, we propose a novel decoder-only Transformer architecture based model, named Unified DLKT ( UniKT ), to learn coherent and unified representations across a wide range of data sources. Specifically, we combine student learning sequences from six educational scenarios and utilize a multi-source encoding to learn unified representations of interactions from mixed data. UniKT is a stack of Transformer decoder layers for handling long-term dependencies among students’ historical interactions and future performance. We evaluate UniKT on six publicly available real-world educational datasets, and experimental results demonstrate that our method outperforms the majority of existing DLKT models in terms of AUC and accuracy. Furthermore, the empirical analysis shows the strong transferability and adaptability of UniKT in learning from multiple sources. To encourage reproducible research, we make our data and code publicly available at https://pykt.org/ .

POI recommendation plays an important role in many applications, such as mobility prediction and location-based advertisements. Existing POI recommendation methods mainly capture the observed patterns in user visits for recommendations, without a comprehensive consideration of the underlying reasons behind the visits. Therefore, different causes of a visit, i.e., users’ interest and geographical context, are entangled. When the underlying causes change (e.g., when a user moves to a new place), the robustness of the recommendations cannot be guaranteed. To address the above challenges, we propose DUIG, a novel user interest and geographical influences disentanglement framework for POI recommendations. We first design a personalized disentanglement strategy to divide check-ins through geographical influence. Specifically, the colliding effect of causality is leveraged to the divide cause-specific check-ins, such that user interest and geographical influence can be properly disentangled in user and POI embeddings. Through this mechanism, even if the underlying reasons that affect a user’s preference change, intervention can be conducted upon the causes to make recommendations generalized to the new scenario. In addition, a geographical-aware negative sampling strategy is proposed to utilize hard negatives to regularize the embedding and disentanglement in the latent space, where a larger sampling probability is introduced for negative samples containing more geographic information. Extensive experiments on two real-world POI recommendation datasets demonstrate the superior performance of DUIG.