Jiayuan Mao 茅佳源

I am a fourth-year PhD student at MIT EECS, advised by Prof. Josh Tenenbaum and Prof. Leslie Kaelbling. Previously, I obtained my bachelor's degree from YaoClass, Tsinghua University.

News
Research Highlights

My long-term research goal is to build machines that can continually learn concepts (e.g., properties, relations, skills, and rules) from their experiences and apply them for reasoning and planning in the physical world.

  • Concept Learning: neuro-symbolic concept learning, grammar-based language learning, pragmatics.
  • Reasoning & Planning: relational and temporal models for reasoning, skill learning, task and motion planning.
  • Scene & Activity Understanding: programmatic scene understanding, activity understanding.
Publications ( show selected / show all by date / show all by topic )

Topics: Concept Learning and Language Acquisition / Reasoning and Planning / Scene and Activity Understanding / Others: Vision / Others: NLP (*/†: indicates equal contribution.)

Learning Rational Subgoals from Demonstrations and Instructions
Zhezheng Luo*, Jiayuan Mao*, Jiajun Wu, Tomás Lozano-Pérez, Joshua B. Tenenbaum, Leslie Pack Kaelbling

AAAI 2023 Comming soon.

On the Expressiveness and Generalization of Hypergraph Neural Networks
Zhezheng Luo, Jiayuan Mao, Joshua B. Tenenbaum, Leslie Pack Kaelbling

LoG 2022 Comming soon.

Sparse and Local Hypergraph Reasoning Networks
Guangxuan Xiao, Leslie Pack Kaelbling, Jiajun Wu, Jiayuan Mao

LoG 2022 Comming soon.

PDSketch: Integrated Domain Programming, Learning, and Planning
Jiayuan Mao*, Tomás Lozano-Pérez, Joshua B. Tenenbaum, Leslie Pack Kaelbling

NeurIPS 2022 Paper / Project Page

HandMeThat: Human-Robot Communication in Physical and Social Environments
Yanming Wan*, Jiayuan Mao*, Joshua B. Tenenbaum

NeurIPS 2022 Datasets and Benchmarks Track Paper / Project Page

CLEVRER-Humans: Describing Physical and Causal Events the Human Way
Jiayuan Mao*, Xuelin Yang*, Xikun Zhang, Noah D. Goodman, Jiajun Wu

NeurIPS 2022 Datasets and Benchmarks Track Paper / Project Page

Translating a Visual LEGO Manual to a Machine-Executable Plan
Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao, Chin-Yi Cheng, Jiajun Wu

ECCV 2022 Paper / Project Page

Programmatic Concept Learning for Human Motion Description and Synthesis
Sumith Kulal*, Jiayuan Mao*, Alex Aiken†, Jiajun Wu†

CVPR 2022 Paper / Project Page

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
Lingjie Mei*, Jiayuan Mao*, Ziqi Wang, Chuang Gan, Joshua B. Tenenbaum

ICLR 2022 Paper / Project Page

Grammar-Based Grounded Lexicon Learning
Jiayuan Mao, Haoyue Shi, Jiajun Wu, Roger P. Levy, Joshua B. Tenenbaum

NeurIPS 2021 Paper Project Page

Temporal and Object Quantification Networks
Jiayuan Mao*, Zhezheng Luo*, Chuang Gan, Joshua B. Tenenbaum, Jiajun Wu, Leslie Pack Kaelbling, Tomer D. Ullman

IJCAI 2021 Paper / Project Page / Code

(First two authors contributed equally; order determined by coin toss.)
Language-Mediated, Object-Centric Representation Learning
Ruocheng Wang*, Jiayuan Mao*, Samuel J. Gershman, Jiajun Wu

ACL 2021 (Findings) Paper / Talk / Project Page

Hierarchical Motion Understanding via Motion Programs
Sumith Kulal*, Jiayuan Mao*, Alex Aiken, Jiajun Wu

CVPR 2021 Paper / Talk / Project Page

Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee K. Wong, Joshua B. Tenenbaum, Chuang Gan

ICLR 2021 Paper / Project Page

Object-Centric Diagnosis of Visual Reasoning
Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David D. Cox, Joshua B. Tenenbaum, Chuang Gan

ArXiv 2020 Paper

Multi-Plane Program Induction with 3D Box Priors
Yikai Li*, Jiayuan Mao*, Xiuming Zhang, William T. Freeman, Joshua B. Tenenbaum, Noah Snavely, Jiajun Wu

NeurIPS 2020 Paper / Video / Project Page

Perspective Plane Program Induction from a Single Image
Yikai Li*, Jiayuan Mao*, Xiuming Zhang, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

CVPR 2020 Paper / Project Page

Visual Concept-Metaconcept Learning
Chi Han*, Jiayuan Mao*, Chuang Gan, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2019 Paper / Project Page / Code

Program-Guided Image Manipulators
Jiayuan Mao*, Xiuming Zhang*, Yikai Li, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

ICCV 2019 Paper / Project Page

(First two authors contributed equally; order determined by coin toss.)
Visually Grounded Neural Syntax Acquisition
Haoyue Shi*, Jiayuan Mao*, Kevin Gimpel, Karen Livescu

ACL 2019 (Best Paper Nomination) Paper / Project Page / Code

Neurally-Guided Structure Inference
Sidi Lu*, Jiayuan Mao*, Joshua B. Tenenbaum, Jiajun Wu

ICML 2019 Paper / Project Page / Code

Unified Visual-Semantic Embeddings:
Bridging Vision and Language with Structured Meaning Representations
Hao Wu*, Jiayuan Mao*, Yufeng Zhang, Weiwei Sun, Yuning Jiang, Lei Li, Wei-Ying Ma

CVPR 2019 (Oral) Paper

The Neuro-Symbolic Concept Learner:
Interpreting Scenes, Words, and Sentences From Natural Supervision
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, Jiajun Wu

ICLR 2019 (Oral) Paper / Project Page / Code / MIT News / MIT Technology Review

Neural Logic Machines
Honghua Dong*, Jiayuan Mao*, Tian Lin, Chong Wang, Lihong Li, Dengyong Zhou

ICLR 2019 Paper / Project Page / Code

Neural Phrase-to-Phrase Machine Translation
Jiangtao Feng, Lingpeng Kong, Po-Sen Huang, Chong Wang, Da Huang, Jiayuan Mao, Kan Qiao, Dengyong Zhou

ArXiv Preprint Paper

Acquisition of Localization Confidence for Accurate Object Detection
Borui Jiang*, Ruixuan Luo*, Jiayuan Mao*, Tete Xiao, Yuning Jiang

ECCV 2018 (Oral) Paper / Code

Learning Visually-Grounded Sementics from Contrastive Adversarial Samples
Haoyue Shi*, Jiayuan Mao*, Tete Xiao*, Yuning Jiang, Jian Sun

COLING 2018 Paper / Code

Universal Agent for Disentangling Environments and Tasks
Jiayuan Mao, Honghua Dong, Joseph J. Lim

ICLR 2018 Paper / Project Page

What Can Help Pedestrian Detection?
Jiayuan Mao*, Tete Xiao*, Yuning Jiang, Zhimin Cao

CVPR 2017 Paper