Jiayuan Mao 茅佳源

Jiayuan Mao is a PhD candidate at MIT EECS, advised by Prof. Josh Tenenbaum and Prof. Leslie Kaelbling. Previously, she obtained her Bachelor's degree from YaoClass, Tsinghua University.

News
Research Highlights

My long-term research goal is to build machines that can continually learn concepts (e.g., properties, relations, skills, rules and algorithms) from their experiences and apply them for reasoning and planning in the physical world. The central theme of my research is to decompose the learning problem into learning a vocabulary of neuro-symbolic concepts. The symbolic part describes their structures and how different concepts can be composed; the neural part handles grounding in perception and physics. I leverage structures to make learning more data-efficient, more compositionally generalizable, and also inference and planning faster.

Neuro-Symbolic Concepts: Representations

How should we represent various types of concepts?

How to capture the programmatic structures underlying these concepts (The Theory-Theory of Concepts)?

Show/Hide Work on Representations

Learning, Reasoning and Planning Algorithms

How can we efficiently learn these concepts from natural supervisions (e.g., language, videos)?

How can we leverage the structures of these concepts to make inference and planning faster?

Show/Hide Work on Algorithms

Publications ( show selected / show all by date / show all by topic )

Topics: Concept Learning and Language Acquisition / Reasoning and Planning / Scene and Activity Understanding
Past topics: Object Detection / Structured NLP (*/†: indicates equal contribution.)

Embodied Agent Interface: A Single Line to Evaluate LLMs for Embodied Decision Making
Manling Li*, Shiyu Zhao*, Qineng Wang*, Kangrui Wang*, Yu Zhou*, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu

NeurIPS 2024 Datasets and Benchmarks Track (Oral)
Paper (Coming Soon) / Project Page / Code / Data

Hybrid Declarative-Imperative Representations for Hybrid Discrete-Continuous Decision-Making
Jiayuan Mao, Joshua B. Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling

WAFR 2024 Paper Preprint / Slides

Agent Workflow Memory
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, Graham Neubig

ArXiv 2024 Paper / Code

Finding Structure in Logographic Writing with Library Learning
Guangyuan Jiang, Matthias Hofer, Jiayuan Mao, Lio Wong, Joshua B. Tenenbaum, Roger P. Levy

CogSci 2024 (Best Undergraduate Student Paper) Paper

Learning Iterative Reasoning through Energy Diffusion
Yilun Du*, Jiayuan Mao*, Joshua B. Tenenbaum

ICML 2024 Paper / Project Page / Code

"Set It Up!": Functional Object Arrangement with Compositional Generative Models
Yiqing Xu, Jiayuan Mao, Yilun Du, Tomás Lozano-Pérez, Leslie Pack Kaelbling, David Hsu

RSS 2024 Paper
Presented at RSS Workshop on Task Specification for General-Purpose Intelligent Robots

Grounding Language Plans in Demonstrations through Counter-factual Perturbations
Yanwei Wang, Tsun-Hsuan Wang, Jiayuan Mao, Michael Hagenow, Julie Shah

ICLR 2024 (Spotlight) Paper / Project Page / Code / MIT News / TechCrunch

Learning Adaptive Planning Representations with Natural Language Guidance
Lio Wong*, Jiayuan Mao*, Pratyusha Sharma*, Zachary S. Siegel, Jiahai Feng, Noa Korneev, Joshua B. Tenenbaum, Jacob Andreas

ICLR 2024 Paper

Learning to Act from Actionless Video through Dense Correspondences
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, Joshua B. Tenenbaum

ICLR 2024 (Spotlight) Paper / Project Page / Code

Learning Planning Abstractions from Language
Weiyu Liu*, Geng Chen*, Joy Hsu, Jiayuan Mao†, Jiajun Wu

ICLR 2024 Paper

What Planning Problem Can A Relational Neural Network Solve
Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum, Leslie Pack Kaelbling

NeurIPS 2023 (Spotlight) Paper / Project Page / Code

What’s Left? Concept Grounding with Logic-Enhanced Foundation Models
Joy Hsu*, Jiayuan Mao*, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2023 Paper / Project Page / Code

Compositional Diffusion-Based Continuous Constraint Solvers
Zhutian Yang, Jiayuan Mao, Yilun Du, Jiajun Wu, Joshua B. Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling

CoRL 2023 Paper / Project Page

Composable Part-Based Manipulation
Weiyu Liu, Jiayuan Mao, Joy Hsu, Tucker Hermans, Animesh Garg, Jiajun Wu

CoRL 2023 Paper / Project Page

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations
Joy Hsu, Jiayuan Mao, Jiajun Wu

CVPR 2023 Paper / Project Page / Code
Presented at CVPR 2023 Workshop On Compositional 3D Vision (Oral)

Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
Renhao Wang*, Jiayuan Mao*, Joy Hsu, Hang Zhao, Jiajun Wu, Yang Gao

ICLR 2023 (Notable Top 25%) Paper / Project Page

Learning Rational Subgoals from Demonstrations and Instructions
Zhezheng Luo*, Jiayuan Mao*, Jiajun Wu, Tomás Lozano-Pérez, Joshua B. Tenenbaum, Leslie Pack Kaelbling

AAAI 2023 Paper / Project Page / Code

DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage
Joy Hsu, Jiayuan Mao, Jiajun Wu

TMLR 2023 Paper / Project Page / Code

On the Expressiveness and Generalization of Hypergraph Neural Networks
Zhezheng Luo, Jiayuan Mao, Joshua B. Tenenbaum, Leslie Pack Kaelbling

LoG 2022 Paper

Sparse and Local Hypergraph Reasoning Networks
Guangxuan Xiao, Leslie Pack Kaelbling, Jiajun Wu, Jiayuan Mao

LoG 2022 Paper / Project Page / Code

PDSketch: Integrated Domain Programming, Learning, and Planning
Jiayuan Mao, Tomás Lozano-Pérez, Joshua B. Tenenbaum, Leslie Pack Kaelbling

NeurIPS 2022 Paper / Project Page / Code

HandMeThat: Human-Robot Communication in Physical and Social Environments
Yanming Wan*, Jiayuan Mao*, Joshua B. Tenenbaum

NeurIPS 2022 Datasets and Benchmarks Track Paper / Project Page / Code

CLEVRER-Humans: Describing Physical and Causal Events the Human Way
Jiayuan Mao*, Xuelin Yang*, Xikun Zhang, Noah D. Goodman, Jiajun Wu

NeurIPS 2022 Datasets and Benchmarks Track Paper / Project Page / Code

Translating a Visual LEGO Manual to a Machine-Executable Plan
Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao, Chin-Yi Cheng, Jiajun Wu

ECCV 2022 Paper / Project Page / Code

Programmatic Concept Learning for Human Motion Description and Synthesis
Sumith Kulal*, Jiayuan Mao*, Alex Aiken†, Jiajun Wu†

CVPR 2022 Paper / Project Page

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
Lingjie Mei*, Jiayuan Mao*, Ziqi Wang, Chuang Gan, Joshua B. Tenenbaum

ICLR 2022 Paper / Project Page

Grammar-Based Grounded Lexicon Learning
Jiayuan Mao, Haoyue Shi, Jiajun Wu, Roger P. Levy, Joshua B. Tenenbaum

NeurIPS 2021 Paper / Project Page / Code

Temporal and Object Quantification Networks
Jiayuan Mao*, Zhezheng Luo*, Chuang Gan, Joshua B. Tenenbaum, Jiajun Wu, Leslie Pack Kaelbling, Tomer D. Ullman

IJCAI 2021 Paper / Project Page / Code

(First two authors contributed equally; order determined by coin toss.)
Language-Mediated, Object-Centric Representation Learning
Ruocheng Wang*, Jiayuan Mao*, Samuel J. Gershman, Jiajun Wu

ACL 2021 (Findings) Paper / Talk / Project Page

Hierarchical Motion Understanding via Motion Programs
Sumith Kulal*, Jiayuan Mao*, Alex Aiken, Jiajun Wu

CVPR 2021 Paper / Talk / Project Page

Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee K. Wong, Joshua B. Tenenbaum, Chuang Gan

ICLR 2021 Paper / Project Page

Object-Centric Diagnosis of Visual Reasoning
Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David D. Cox, Joshua B. Tenenbaum, Chuang Gan

ArXiv 2020 Paper

Multi-Plane Program Induction with 3D Box Priors
Yikai Li*, Jiayuan Mao*, Xiuming Zhang, William T. Freeman, Joshua B. Tenenbaum, Noah Snavely, Jiajun Wu

NeurIPS 2020 Paper / Video / Project Page

Perspective Plane Program Induction from a Single Image
Yikai Li*, Jiayuan Mao*, Xiuming Zhang, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

CVPR 2020 Paper / Project Page

Visual Concept-Metaconcept Learning
Chi Han*, Jiayuan Mao*, Chuang Gan, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2019 Paper / Project Page / Code

Program-Guided Image Manipulators
Jiayuan Mao*, Xiuming Zhang*, Yikai Li, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

ICCV 2019 Paper / Project Page

(First two authors contributed equally; order determined by coin toss.)
Visually Grounded Neural Syntax Acquisition
Haoyue Shi*, Jiayuan Mao*, Kevin Gimpel, Karen Livescu

ACL 2019 (Best Paper Nomination) Paper / Project Page / Code

Neurally-Guided Structure Inference
Sidi Lu*, Jiayuan Mao*, Joshua B. Tenenbaum, Jiajun Wu

ICML 2019 Paper / Project Page / Code

Unified Visual-Semantic Embeddings:
Bridging Vision and Language with Structured Meaning Representations
Hao Wu*, Jiayuan Mao*, Yufeng Zhang, Weiwei Sun, Yuning Jiang, Lei Li, Wei-Ying Ma

CVPR 2019 (Oral) Paper
Presented at NAACL 2019 SpLU-RoboNLP

The Neuro-Symbolic Concept Learner:
Interpreting Scenes, Words, and Sentences From Natural Supervision
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, Jiajun Wu

ICLR 2019 (Oral) Paper / Project Page / Code / MIT News / MIT Technology Review

Neural Logic Machines
Honghua Dong*, Jiayuan Mao*, Tian Lin, Chong Wang, Lihong Li, Dengyong Zhou

ICLR 2019 Paper / Project Page / Code

Neural Phrase-to-Phrase Machine Translation
Jiangtao Feng, Lingpeng Kong, Po-Sen Huang, Chong Wang, Da Huang, Jiayuan Mao, Kan Qiao, Dengyong Zhou

ArXiv Preprint Paper

Acquisition of Localization Confidence for Accurate Object Detection
Borui Jiang*, Ruixuan Luo*, Jiayuan Mao*, Tete Xiao, Yuning Jiang

ECCV 2018 (Oral) Paper / Code

Learning Visually-Grounded Sementics from Contrastive Adversarial Samples
Haoyue Shi*, Jiayuan Mao*, Tete Xiao*, Yuning Jiang, Jian Sun

COLING 2018 Paper / Code

Universal Agent for Disentangling Environments and Tasks
Jiayuan Mao, Honghua Dong, Joseph J. Lim

ICLR 2018 Paper / Project Page

What Can Help Pedestrian Detection?
Jiayuan Mao*, Tete Xiao*, Yuning Jiang, Zhimin Cao

CVPR 2017 Paper