Reinforcement Learning for Robotics

Chenhao Li

Master Student in Robotics, Systems and Control

ETH Zurich

Biography

I am currently a master’s student in Robotics, Systems and Control at ETH Zurich, Switzerland. Before that, I received my bachelor’s degree in Mechanical Engineering with Excellent Graduate Honor as a two-time National Scholarship winner from Tongji University, Shanghai, China.

Since January 2023, I have been working on structured skill representation for humanoids at the Biomimetic Robotics Lab at Massachusetts Institute of Technology with Prof. Sangbae Kim. Prior to that, I was engaged in agile skill imitation from limited demonstrations for legged systems with Dr. Georg Martius at the Autonomous Learning Group at Max Planck Institute for Intelligent Systems (MPI-IS), Tübingen, Germany.

My knowledge of legged robotic control using reinforcement learning was developed and refined during my work on hierarchical skill integration for ANYmal with Prof. Marco Hutter at Robotic Systems Lab (RSL), ETH Zurich, Switzerland. Before that, I was also honored to work on AgnathaX with Prof. Auke Ijspeert at Biorobotics Laboratory (BioRob), EPFL, Switzerland.

My research interests mainly focus on incorporating cutting-edge machine learning approaches into robotic control, especially for developing agile and robust skills for legged systems using reinforcement learning techniques.

For more information, please refer to my latest Resumé.

Interests

Artificial Intelligence
Reinforcement Learning
Legged Locomotion

Education

MSc in Robotics, Systems and Control

ETH Zurich
BEng in Mechanical Engineering

Tongji University

News

2023

Mar 02: I received the ETH AI Center Doctoral Fellowship.

I am excited to share that I received the ETH AI Center Doctoral Fellowship at ETH Zurich, Switzerland. ETH AI Center Fellowship is awarded to doctoral students, where young researchers are co-supervised by two faculty members from different departments within ETH Zurich, thus fostering a culture of interdisciplinary exchange and collaboration. I am extremely honored to receive this offer from Prof. Andreas Krause and Prof. Marco Hutter.

Feb 16: Our work on Solo robots at MPI-IS was reported by German TV network.

My work on Solo robots during my internship at the Max Planck Institute for Intelligent Systems was covered in NANO of 3sat. In the coverage, Dr. Georg Martius explained how autonomous learning systems can obtain extraordinary intelligence from interactions with the environment.

Feb 04: I received admissions to the Ph.D. program at MIT EECS and other prominent institutes.

I am excited to share that I received admission invitation to the Ph.D. program at the department of Electrical Engineering and Computer Science (EECS) and Mechanical Engineering (MechE) at Massachusetts Institute of Technology (MIT), Cambridge, United States. I was also lucky to be admitted to the Robotics Institute at Carnegie Mellon University (CMU), Pittsburgh, United States. I am honored to view these as a recognition and acknowledgement of my work during my master's studies.

Jan 16: Our work was accepted to ICRA 2023.

Our work Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions was accepted by IEEE International Conference on Robotics and Automation (ICRA). ICRA 2023 will be held in London, United Kingdom from May 29 to June 2, 2023.

Jan 09: I started my visit at Biomimetic Robotics Lab at MIT.

Starting from January, I am honored to work with Prof. Sangbae Kim at Biomimetic Robotics Lab at Massachusetts Institute of Technology (MIT), Cambridge, United States. My main research topic focuses on developing latent skill representation for MIT Humanoid robot. I am happy to collaborate with Steve Heim and Elijah Stanger-Jones on this exciting topic.

2022

Dec 19: Great time at CoRL 2022 and satisfying results for my first publication.

It was wonderful to have met all the great names and my future collaborators and advisors during Conference on Robot Learning (CoRL) 2022 in Auckland, New Zealand. I am excited to see that our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations has been acknowledged by the greatest roboticists in the field and was announced into the finalist of the best paper award. Thanks to the team Marin Vlastelica, Sebastian Blaes, Jonas Frey, Felix Grimminger and Georg Martius, who made this achievement possible.

Dec 16: Our work made it into the finalist of the best paper award at CoRL 2022, Auckland, New Zealand.

It was announced during Conference on Robot Learning (CoRL) 2022 in Auckland, New Zealand that our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations made it into the finalist of the best paper award. As one of the three nominated papers in this list, our work has received great acknowledgment from the community.

Dec 02: Our work was presented to the Machines in Motion Laboratory, New York University, United States.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was presented to the scholars at the Machines in Motion Laboratory led by Prof. Ludovic Righetti at New York University. The Machines in Motion Laboratory focuses on understanding the fundamental principles for robot locomotion and manipulation that will endow robots with the robustness and adaptability necessary to efficiently and autonomously act in an unknown and changing environment.

Nov 25: I will join the Biomimetic Robotics Lab at MIT for my master thesis.

I am glad to share that I will join the Biomimetic Robotics Lab led by Prof. Sangbae Kim at Massachusetts Institute of Technology (MIT) to complete my master thesis later this year. The Biomimetic Robotics Lab focuses on designing and controlling robots using insights taken from the natural world. I will be working with the MIT Humanoid on human demonstration imitation.

Nov 22: Our work was accepted to CoRL 2022 Learning for Agile Robotics workshop.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was accepted to the Learning for Agile Robotics workshop at Conference on Robot Learning (CoRL) 2022. This workshop invites researchers working on making agile robots. The goals include fostering collaboration, understanding the current limitations and inefficiencies, learning common ideas on modern ML based approaches, etc.

Nov 21: Our work made it into the Video Friday collection at IEEE Spectrum again.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was covered by Video Friday collection at IEEE Spectrum. Video Friday is a weekly selection of state-of-the-art robotics videos, collected by scholars at IEEE Spectrum Robotics. This time, our overview video was selected and well acknowledged.

Sep 15: Our work was submitted to IEEE International Conference on Robotics and Automation (ICRA) 2023.

During the second three months of my internship, I accomplished the work Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions under the guidance by Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey and Dr. Georg Martius. In this work, Solo obtained diverse skills by imitating unlabeled, mixed reference motions.

Sep 10: Our work was accepted to CoRL 2022 for an oral presentation with best paper award nomination.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was accepted by Conference on Robot Learning (CoRL) 2022 for an oral presentation with best paper award nomination. CoRL 2022 will be held in Auckland, New Zealand from December 14 to 18, 2022.

Aug 10: Our work was presented to the Robotic Systems Lab, ETH Zurich, Switzerland.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was presented to the scholars at the Robotic Systems Lab (RSL), ETH Zurich during the research presentation. In this weekly event, internal and external reseachers are invited to inform their state-of-the-art discoveries in fields including robotic system integration and control methods.

Jul 02: Our work was presented to the public at the Science & Innovation Days, Tübingen.

The experiments of our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations were presented to the public at the Tübingen Science & Innovation Days. In this event, research institutes including the Max Planck Institute for Intelligent Systems open their doors to the interested public and present their research.

Jul 01: Our work made it into the Video Friday collection at IEEE Spectrum.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was covered by Video Friday collection at IEEE Spectrum. Video Friday is a weekly selection of state-of-the-art robotics videos, collected by scholars at IEEE Spectrum Robotics. Among our selected videos, SOLOBACKFLIP gained particular popularity.

Jun 29: Our work was presented to scholars from the International Max Planck Research School for Intelligent Systems.

Our work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations was presented to the evaluation committee and PhD scholars from the International Max Planck Research School for Intelligent Systems (IMPRS) during the on-site evaluation.

Jun 15: Our work was submitted to Conference on Robot Learning (CoRL) 2022.

During the first three months of my internship, I accomplished the work Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations under the guidance by Marin Vlastelica, Sebastian Blaes, Jonas Frey, Felix Grimminger and Dr. Georg Martius. In this work, Solo developed highly dynamic skills by imitating rough, partial, human-demonstrated reference motions.

Apr 01: I started my internship at Autonomous Learning Group at Max Planck Institute for Intelligent Systems.

Starting from April, I am honored to work with Dr. Georg Martius at Autonomous Learning Group at Max Planck Institute for Intelligent Systems (MPI-IS), Tübingen, Germany. My main research topic focuses on agile skill development and style diversification for Solo robot. This work is closely collaborated with Robotic Systems Lab (RSL), ETH Zurich, Switzerland.

Mar 30: My semester project at Robotic Systems Lab (RSL) finished with successful policy deployment on the real system.

By employing a novel compositional control structure, ANYmal learned to achieve full-pose trajectory tracking in a global coordinate system. A hierarchical system pipeline was introduced, where the high-level policy proceeds global target information and outputs low-level composition signals and commands with trivial regularization.

2021

Oct 01: I started my semester project at Robotic Systems Lab (RSL), ETH Zurich, Switzerland.

From October, I am honored to work on a semester project on the skill integration for ANYmal with hierarchical reinforcement learning at Robotic Systems Lab (RSL) directed by Prof. Marco Hutter. The main research topic focuses on extending locomotion and manipulation capabilities for legged systems by leveraging approriate combination of acquired skills. This project is mainly supervised by Yuntao Ma, Takahiro Miki and Mayank Mittal.

Sep 01: I started my teaching assistant position for Linear & Combinatorial Optimization at ETH Zurich, Switzerland.

After having achieved top grades at the course Linear & Combinatorial Optimization (L&CO, 11 ECTS) instructed by Prof. Rico Zenklusen at ETH Zurich last semester, I was honored to take a teaching assitant position for the new semester. Among my responsibilities, I enjoyed delivering lectures the most.

Jul 9: Wonderful week at ETH Robotics Summer School in Wangen an der Aare, Switzerland.

From July 2 to 9, I was honored to participate in the ETH Robotics Summer School directed by ETH RobotX initiative. During this event, a broad scope of components of autonomous mobile robots including state estimation, trajectory optimization, envinronment mapping, path planning, and artifact detection were introduced and implemented on a wheeled platform, SuperMegaBot. On the last day, we successfully completed an autonomous rescue challenge at the test site.

2020

Sep 01: I started my master's program in Robotics, Systems and Control at ETH Zurich, Switzerland.

After receiving offers from top-class universities including University of California, Los Angeles (UCLA), Johns Hopkins University (JHU), École polytechnique fédérale de Lausanne (EPFL), Imperial College London, etc, I decided to pursue my future endeavors in Robotics, Systems and Control under the guidance by Prof. Marco Hutter at ETH Zurich, Switzerland.

Jun 15: I was awarded Excellent Graduate of Shanghai.

Together with 11 other students from the School of Mechanical Engineering, it was my honor to be granted this highest honor for all undergraduates of all majors from more than ten universities in Shanghai this year.

Feb 01: My research project at Biorobotics Laboratory (BioRob) finished with successful hardware test.

My work on Analysis, Design and Implementation of a Bio-inspired Passive Tail for Amphibious Robots concluded with a novel development of an appropriate chordwise stiffness distribution along a bio-inspired tail for AgnathaX which maximizes the propulsion generated under a combined movement of both passive heaving and pitching.

2019

Sep 01: I started my research project at Biorobotics Laboratory (BioRob) at École polytechnique fédérale de Lausanne (EPFL), Switzerland.

I am honored to work as a research assistant on the design of a bio-inspired passive tail for amphibious robots at Biorobotics Laboratory (BioRob) directed by Prof. Auke Ijspeert during the last year of my bachelor's study. This project is mainly supervised by Laura Paez.

2018

Nov 16: I was awarded my second National Scholarship for my study at Tongji University, Shanghai, China.

The Ministry of Education of China funded the China National Scholarship to award outstanding full-time undergraduates for their excellent academic records. Being among top 1% of the students of the same year again, I am honored to receive this award a second time.

Oct 01: I finished my research internship at Advanced Multifunctional and Multiphysics Metamaterials Lab (AM3L) of the Department of Bioresource Engineering of McGill University, Montréal, Canada.

From July to October, I was honored to be selected by Mitacs Globalink Research Internship program and work on Design and 3D Printing of Lightweight Advanced Energy Harvesters Made of Smart Materials under the supervision of Prof. Abdolhamid Akbarzadeh. The research was devoted into reinforcing mechanical properties of 3D printing materials by affiliating pulverized wood particles with melted polymer through a novel process of extrusion.

Apr 01: I was awarded Meritorious Winner in 2018 Mathematical Contest in Modeling.

I am honored to receive my first international award in Mathematics.

2016

Nov 16: I was awarded my first National Scholarship for my study at Tongji University, Shanghai, China.

The Ministry of Education of China funded the China National Scholarship to award outstanding full-time undergraduates for their excellent academic records. Being among top 1% of the students of the same year, I am honored to receive this award.

Featured Publications

Chenhao Li, Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey, Georg Martius

September, 2022 In ICRA

Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions

In this work, we propose a cooperative generative adversarial method for obtaining controllable skill sets from unlabeled datasets containing diverse state transition patterns.

Chenhao Li, Marin Vlastelica, Sebastian Blaes, Jonas Frey, Felix Grimminger, Georg Martius

June, 2022 In CoRL Best Paper Award Finalist

Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations

In this work, we propose a generative adversarial method for inferring reward functions from partial and potentially physically incompatible demonstrations for successful skill acquirement where reference or expert demonstrations are not easily accessible.

Projects

Legged Locomotion with Graph Neural Network Policies

Proposing structured control policies for locomotion tasks via graph neural networks based on the robot configuration morphology.

Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions

Proposing a cooperative generative adversarial method for obtaining controllable skill sets from unlabeled datasets containing diverse state transition patterns.

Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations

Proposing a generative adversarial method for inferring reward functions from partial and potentially physically incompatible demonstrations for successful skill acquirement where reference or expert demonstrations are not easily accessible.

Object Manipulation via Hierarchical Reinforcement Learning Control

Employing a hierarchical control method to achieve complex high-level tasks by integrating locomotion and manipulation skills.

Autonomous Pose Tracking with Compositional Reinforcement Learning Policies

Employing a novel compositional control structure to allow legged systems to achieve full-pose trajectory tracking in a global coordinate system by integrating learned skills.

Hierarchical Deep Reinforcement Learning for Legged Robot Navigation

Learning to utilize learned low-level skills in high-level tasks by considering only high-level objectives.

Actor-Critic Reinforcement Learning for a Lunar Lander

Extending vanilla policy gradients with advantage estimation.

Gaussian Process Regression for Groundwater Pollution Prediction

Predicting model posterior distribution with Gaussian Process regression from scratch, which is then applied to an inference problem based on space data.

Hyperparameter Tuning with Constrained Bayesian Optimization

Joint training of model and objective with appropriate acquisition function and constraint satisfaction.

Predicting Uncertainty with Bayesian Neural Networks on MNIST Dataset

Comparing classification uncertainty prediction with DenseNets and Bayesian Neural Networks.

Learning image embeddings and distributions given relative distance labels.

Food Taste Similarity Prediction Based on Images and Human Judgments

Medical Events Prediction with Missing Features and Imbalanced Classification

Predicting orders of medical tests, sepsis, and key vital signs based on incomplete, imbalanced data.

State Estimation with Hybrid Extended Kalman Filter for Boat Pose Tracking

Tracking system states with nonlinear continuous-time dynamics with Hybrid Extended Kalman Filter.

State Estimation with Particle Filter for Mobile Robot Pose Tracking

Tracking system states with general nonlinear system and general noise distributions with Particle Filter.

Wheeled Platform Integration for Search and Rescue Applications

Developing algorithmic components for a wheeled autonomous mobile platform that operates autonomously in search and rescue scenarios.

Boids Behavior Control

Developing control algorithms that simulate the flocking behavior of Boids.

Constrained Rigid Body Simulation and Impulse-based Collision Modeling

Simulating rigid body dynamics by solving the equations of motion and impulse-based collisions with restitutional contact.

Inverse Kinematics Locomotion Control for a Legged Robot

Finding desired joint angles in trajectory tracking problems with inverse kinematic control.

Locomotion, Perception, Localization and Path Planning for an Autonomous Mobile Robot

Developing algorithmic components for autonomous mobile robot integration.

Mechanical Simulation and Soft Manipulation of a Soft Bar with the Finite Element Method

Applying the Finite Element Method to simulate the mechanical properties of a soft bar applied with constraints and external forces.

Topological Network Properties of the European Football Loan System

Analyzing topological structures of a graph network in view of different time spans for different local measures.

Trajectory Optimization of a Spaceship with Direct Single Shooting and Direct Transcription

Optimizing trajectories of a spaceship hitting a target and orbiting a planet in a space gravity field.

Nonlinear Quadrotor Full Position Control

Developing nonlinear control of position and yaw angle for quadrotor systems.

Model Predictive Control for Vaccination Center Climate Regulation

Regulating continuous-time linear systems with a variety of Model Predictive Control (MPC) controllers.

Solving Bellman Equations in a Stochastic Shortest Path Problem of Package Delivery

Comparing computational complexity of deriving the optimal cost and control policy using value iteration, policy iteration and linear programming algorithms in a discrete infinite horizon setting.

Analysis, Design and Implementation of a Bio-inspired Passive Tail for Amphibious Robots

Developing an appropriate chordwise stiffness distribution along a bio-inspired tail for amphibious robot systems which maximizes the propulsion generated under a combined movement of both passive heaving and pitching.

Design and 3D Printing of Lightweight Advanced Energy Harvesters Made of Smart Materials

Reinforcing mechanical properties of 3D printing materials by affiliating pulverized wood particles with melted polymer through a novel process of extrusion.

Experience

Visiting Researcher

Massachusetts Institute of Technology

Jan 2023 – Present Cambridge, Massachusetts, United States

Latent skill development for humanoids under the supervision of Prof. Sangbae Kim.

Research Intern

Max Planck Institute for Intelligent Systems

Apr 2022 – Dec 2022 Tübingen, Germany

Agile skill development and style diversification for quadrupedal robots under the supervision of Dr. Georg Martius.

Teaching Assistant

ETH Zurich

Sep 2021 – Jan 2022 Zurich, Switzerland

Recitation delivery for Linear & Combinatorial Optimization (Mathematical Optimization, 11 ECTS) instructed by Prof. Rico Zenklusen.

Research Assistant

École polytechnique fédérale de Lausanne (EPFL)

Sep 2019 – Jan 2020 Lausanne, Switzerland

Analysis, design and implementation of a bio-inspired passive tail for amphibious robots under the supervision of Prof. Auke Ijspeert.

Research Assistant

McGill University

Jul 2019 – Sep 2020 Montréal, Canada

Reinforcing the mechanical properties of 3D printing materials by affiliating pulverized wood particles with melted polymer through a novel process of extrusion under the supervision of Prof. Abdolhamid Akbarzadeh.