Huan Ling

I am a Research Manager and Senior Research Scientist at NVIDIA Spatial Intelligence (TorontoAI) Lab, leading a group focusing on generative world models.

Our mission is to develop and open-source high-fidelity generative foundation models and tools that enable new forms of generation and interaction for both physical and digital creative content.

Contact me at: linghuan at cs.toronto.edu

Github / Google Scholar / X

• Oct. 2025: World Foundation Models for Physical AI Applications @ BAIR
• June 2025: Cosmos-Drive-Dreams, Keynote talk @ CVPR'25, 6th Embodied AI Workshop (EAI)
• Oct. 2023: See Through the Eyes of Generative Models, UBC
• July 2023: Vision generation and perception using Diffusion Models, ByteDance AD Core(Slides)
• June 2023: Image, Video and 3D Content Creation with Diffusion Models, BAAI 2023 (Slides)
• June 2023: Align your Latents: VideoLDM, Shanghai AI Lab (Slides)
• July 2021: GANs for 2D Vision Perception, Walmart CV Conf (Slides)
• June 2021: GANs for 2D Vision Perception, IIIS, Tsinghua University (Slides)

Open-sourced world foundation models and tools.

A framework for temporal reasoning that enables advanced image editing and world simulation through understanding time-based relationships.

A world model based synthetic data generation (SDG) pipeline designed to enhance downstream tasks for autonomous vehicles.

Large-scale multimodal control for conditional world generation.

NVIDIA's Cosmos product and open-source models.

Media: Two Minute Papers; 机器之心

Full list at Google Scholar.

Generative Models:

A generative 3D scene reconstruction method that uses video diffusion model self-distillation to create high-quality 3D representations.

Enhancing 3D reconstructions and novel-view synthesis via single step diffusion inference.

A neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework.

A generative video model with precise Camera Control and temporal 3D Consistency with a 3D cache

Media: Two Minute Papers

A scalable and expressive framework for high-quality dynamic 3D scene reconstruction.

We propose a framework that aligns dynamic 3D Gaussians with text-driven 4D generations.

We present a latent diffusion framework for high-resolution video synthesis. A follow-up open sourced model, Stable Video Diffusion (SVD), is available at huggingface, featuring enhanced datasets and fine-tuned results.

Media: Two Minute Papers

EditGAN enables fine-grained and high-quality semantic edits to images by directly manipulating the latent space of GANs with explicit control over object-level attributes.

Media: Two Minute Papers

Generative Representation Learning:

A novel 3D object detection approach that leverages geometry-aware features derived from diffusion models for more robust 3D understanding.

We propose a self-supervised pretraining framework that uses generative models to supervise image encoders without requiring labeled data.

We extend DatasetGAN to synthesize large-scale datasets like ImageNet with dense pixel-wise annotations, significantly reducing the manual labeling burden.

DatasetGAN leverages the rich latent space of GANs to synthesize annotated datasets with minimal human effort, enabling pixel-level labeling from a few examples.

We introduce a variational framework for completing partially occluded objects, producing plausible and diverse amodal completions for scene understanding.

3D Vision:

GANverse3D

We unify GANs and differentiable rendering for interpretable 3D reconstruction and inverse graphics from single images using GAN priors and neural rendering techniques.

Two-minute paper

We propose DIB-R, a differentiable renderer that enables training of 3D object predictors end-to-end with supervision from 2D images alone.

Interactive Annotation:

ScribbleBox is a practical system that allows users to quickly annotate video object segmentations with scribbles, enabling high-quality masks with minimal effort.

We introduce Curve-GCN, a real-time annotation tool that models object contours as closed curves and uses graph convolution to refine annotations interactively.

Polygon-RNN++ is an annotation system that speeds up segmentation mask creation by predicting object boundaries with minimal clicks via recurrent polygon prediction.

Fun note from year 2023: Yes, we took a bite of RLHF back in 2017 :)

We introduce an early reinforcement learning with human feedback (RLHF) approach for image captioning, where a model learns to improve descriptions through natural language feedback rather than numerical rewards.

I borrowed the template from ✩, ✩,✩,✩.