• Home
  • Search
  • Human-centric 3D representation learning
  • https://doi.org/10.32657/10356/184558Copy DOI Icon

Human-centric 3D representation learning

  • Jan 1, 2025
  • Fangzhou Hong
Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Understanding the world in three dimensions has long been a scientific challenge with significant practical implications. In the era of deep learning, researchers have made substantial progress in developing 3D representations using deep neural networks. Among the myriad of entities in our complex environment, humans stand out due to their general significance in science and their specific relevance in various applications. This thesis focuses on human-centric 3D representation learning within the framework of deep learning. Specifically, we explore three key areas: human-centric 3D perception, human reconstruction, and 3D human generation. The thesis presents five studies that collectively address these topics. In the realm of human-centric 3D perception, we introduce a versatile multi-modal pre-training approach. By harnessing the diverse modalities of human data, such as RGB images, depth, and 2D keypoints, we present a general framework HCMoCo for effective human-centric representation learning. This framework achieves state-of-the-art performance across four human perception tasks, including DensePose prediction, human parsing, and 3D keypoint prediction. In the area of human reconstruction, we investigate garment reconstruction from 4D point clouds of dressed individuals. To address the ambiguities inherent in 2D images, we propose a principled framework called Garment4D, which enables separable and interpretable garment reconstruction. Notably, Garment4D can effectively reconstruct and model the non-rigid deformations of loose garments (e.g., skirts) that do not share the same topology as the human body. We present two distinct approaches to 3D human generation. In our first work, AvatarCLIP, we generate and animate avatars based on text descriptions of body shapes, appearances, and motions. By leveraging differentiable rendering and large- scale vision-language pre-trained models, we achieve avatar and motion synthesis without the need for supervised training or paired data. To further enhance the quality of the generated avatars, we introduce a second approach, EVA3D, which is a high-quality unconditional 3D human generative model that requires only 2D image collections for training. We design an efficient compositional human NeRF representation to facilitate high-resolution 3D human sampling and rendering, which is employed in adversarial training. Finally, by harnessing the capabilities of large language models (LLMs), we explore 3D human representation learning with a focus on human motion and the integration of perception, reconstruction, and generation. We introduce EgoLM, a versatile framework for understanding egocentric motion using multi-modal data. This framework incorporates rich contextual information from egocentric videos and motion sensors provided by wearable devices. EgoLM unifies various motion learning tasks, including motion understanding from video and motion data, as well as motion tracking and generation from text or sparse sensor input. Additionally, it enables a novel task unique to wearable devices: generating text descriptions from sparse sensor data. By investigating 3D human representation learning from three distinct perspectives, we have attained a comprehensive understanding of humans that spans both high-level semantics and low-level geometry. This thesis offers a cohesive exploration of human-centric 3D representations, contributing significantly to the field of human-centric vision.

Similar Papers
  • Research Article

MirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing

  • Jan 01, 2026
  • IEEE Transactions on Software Engineering
  • Shiwen Ou +8
  • Book Chapter
  • Citations44

Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations

  • Jan 01, 2022
  • Atsuhiro Noguchi +3
  • Conference Article
  • Citations24

Learning Deep Network for Detecting 3D Object Keypoints and 6D Poses

  • Jun 01, 2020
  • Wanqing Zhao +5
  • Book Chapter
  • Citations2

A Comparative Study on Construction of 3D Objects from 2D Images

  • Jan 01, 2021
  • Mohan Mahanty +5
  • Research Article

How to Enhance Causal Discrimination of Emotional Utterances: A Case on LLMs

  • Oct 01, 2025
  • IEEE Transactions on Affective Computing
  • Xinyu Yang +3
  • Research Article

Generative Artificial Intelligence in Urban Design: A Review of Recent Applications

  • Dec 01, 2025
  • Landscape Architecture
  • Qiyuan Hong +2
  • Research Article

Semantic Similarity Learning for Medical Literature: A Multimodal Framework Combining LLM and Deep Metric Learning

  • Apr 10, 2026
  • International Journal of Drug Delivery Technology
  • Deepika A +1
  • Conference Article
  • Citations88

Vision Transformer for NeRF-Based View Synthesis from a Single Input Image

  • Jan 01, 2023
  • Kai-En Lin +5
  • Research Article

Website Owner Identification through Multi-level Contrastive Representation Learning

  • Oct 18, 2025
  • ACM Transactions on Knowledge Discovery from Data
  • Cheng Tu +6
  • Research Article

Rooftop wind field reconstruction using sparse sensors: From deterministic to generative learning methods

  • Mar 01, 2026
  • Building and Environment
  • Yihang Zhou +4
  • Research Article
  • Citations23

Enhanced Structure-Based Prediction of Chiral Stationary Phases for Chromatographic Enantioseparation from 3D Molecular Conformations.

  • Feb 03, 2024
  • Analytical Chemistry
  • Yuhui Hong +3
  • Conference Article
  • Citations5

Representation Learning for Wafer Pattern Recognition in Semiconductor Manufacturing Process

  • Feb 20, 2023
  • Scunghwan Song +1
  • Research Article
  • Citations23

Compositional Scene Representation Learning via Reconstruction: A Survey.

  • Oct 01, 2023
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Jinyang Yuan +3
  • Conference Article
  • Citations687

Learning Continuous Image Representation with Local Implicit Image Function

  • Jun 01, 2021
  • Yinbo Chen +2
  • Research Article

R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generation.

  • Jan 01, 2026
  • IEEE journal of biomedical and health informatics
  • Xiao Wang +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.