Course in Robot (real-world-agent) Learning


Learning from images	Learning reward functions	Learning in the real world

I teach a course on machine learning for the real world. This course focuses on deep reinforcement learning methods and their application to control real world systems (robotics, etc). Here is a link to where you can find the course offered on the DIRO web page as IFT 6163.

Learning methods such as deep reinforcement learning have shown success in solving simulated planning and control problems but struggle to produce diverse, intelligent behaviour on systems that interact in the real world (robots). This class aims to discuss these limitations and study methods to overcome them and enable agents capable of training autonomously, becoming learning and adapting systems that require little supervision. By the end of the course, each student should have a solid grasp of different techniques to train agents to accomplish tasks in the real world. These techniques covered in the course include but are not limited to reinforcement learning, batch RL, multi-task RL, model-based RL, Sim2Real, hierarchical RL, goal-conditioned RL, multi-Agent RL, the fragility of RL, meta-level decision making and learning reward functions.

Target and objectives of the course

Learn the fundamental concepts of machine learning for robotics applications. Such concepts are considered advanced and require a good foundation in machine learning, deep learning and reinforcement learning. This involves:

Becoming familiar with the main types of machine learning models for a control policy (from model-based to model-free)
Developing the ability to read research articles, contextualize them and develop a critical mind;
Develop presentation skills;
Develop their research autonomy in machine learning.
Develop skills related to the strengths and weaknesses of current machine learning methods when being applied to real-world problems.

Prerequisites of the course

You must have completed or are taking in parallel one of the following courses (or equivalent).

The course will also use Python heavily. I will assume familiarity with linear algebra, probability, statistics, planning, optimization, and operating systems (e.g. multi-threading and memory management). You also need to be able to read and understand research papers from NeurIPS, RSS, ICRA, CoRL, and ICLR.

The objectives of the assignments and final projects will be:

Software knowledge: Learning about the available software that is used for deep reinforcement learning.
Analysis skills: Collecting proper statistics of results and using server computers and docker to reproduce experiments and validate results.
Proposal of a new idea to explore for the final project based on the lectures;
Learn how to understand the potential of a method in the real world (positive or negative, social and environmental);
Exploration with final project: More free-form investigation of advanced topics from class that produces reusable code, reproducible results and a written report.
Estimation of the impact that the new idea could have in an industrial context;

The final project will be achieved as a team of two. Research in academia and industry involves working with others to achieve research goals. This project will evaluate the students’ ideas, research process understanding, teamwork, and presentation.

Regarding the lectures, the instructor will provide live lectures on the topics each week, focusing on their understanding, application, and limitations.

EVALUATIVE APPROACH AND WEIGHTING (indicative only)

40%: Programming assignments. (This is needed to provide skills necessary to perform a good final project in the class)
10%: Class participation and discussing readings.
15%: Midterm on concepts
35%: Final project

Programming Assignments

The programming assignments cover topics that are important to perform research on combining machine learning and robots. They will cover:

Behaviour cloning and imitation learning
Model-free (PPO) and model-based RL methods (DDPG or PETS).
Exploration and pretraining methods, such as HRL and Goal conditioned RL.
Safe Exploration
Learning reward functions (un/smi-supervised RL)

The assignments are also designed to familiarize students with the software needed to perform research:

Deep learning libraries, such as pytorch or tensorflow
Hardware constraints when working with real robots (power, compute, mechanical limits)
Distributed computing for running proper experiments
Visualization and analysis (the most important part)

Final Project

The final project is designed to let students spread their wings and apply the learned skills in an area of interest of the students while digging deeper into the concepts. Your project does not need to use real robot hardware but there are options to get hardware for this class. Each project will start with a proposal that will ensure the right scope for each project. The projects will be in groups of 2 or 3 students. However, to provide some ideas on the scope, a few examples are given below.

Re-implement a method in a new deep learning framework
Re-implement a method in a paper that did not release code (shame on them)
Choose a robot to get and train a model to make it solve a task, like walking.
Investigate a new method that improves exploration
Investigate a new method for learning an improved representation for learning and exploration.
Study Sim2Real by learning and transferring policies to another simulation or real robot hardware.
Perform a literature review of Sim2Real papers.

One of the broad goals of the class project is also to refine your skills at defining reasonable projects that can be completed and are of sufficient quality.

Real Robot Hardware

In order to use real hardware for your class project, you need to include the requested hardware in the project proposal. The course has money allocated to purchase hardware for projects.

Some robot ideas for projects

Resources

Some content related to the course that will be helpful to review.