2 posts tagged with "projects"

Recently I gained interest in Reinforcement Learning (RL) for robotics, especially after working on AI orchestration during my internship. After trying a few things, I found that the best way to start is to use the Gymnasium library for creating environments and the Stable Baselines3 library for training RL agents.

These two open-source libraries are perfectly suited for beginners in RL as you can quickly get some results without having to implement everything from scratch and then implement your own environments and agents as you progress. But the best part is that they have out-of-the-box support for robotics simulation environments in Mujoco which is exactly what I was looking for.

If you want to try it out, here is a dependency list for you favorite package manager:

dependencies = [
    "stable-baselines3[extra] (>=2.6.0,<3.0.0)",
    "gymnasium[other] (>=0.29.1,<1.2.0)",
    "box2d (>=2.3.10,<3.0.0)",
    "swig (>=4.3.1,<5.0.0)",
    "gymnasium-robotics (>=1.3.1,<2.0.0)",
    "sb3-contrib (>=2.6.0,<3.0.0)",
]

If this didn't convince you yet, here is what I achieved after a few tries with the Double Pendulum environment and the CrossQ algorithm :

This is a simple example, but as someone who is used to working with classical control methods, I found it very interesting to see how easy it is to get a working RL agent that can control a complex system like this one.

This post is an update on my current projects and what I'm working on. I might do similar posts in the future (mostly to keep track of my projects), so stay tuned!

What I've been up to​

French Robotic Cup 2025​

A few weeks ago I participated in the French Robotic Cup 2025 with my team, Metal Pipe. Our goal was to build robots as simple as possible due to our limited budget and time. Some of our team members were new to robotics and some were working from the other side of the country so we had to make sure that the robot was easy to build and maintain. We ended up building a differential drive robot with a precise control loop but we were limited by our hardware both in terms of mechanics and electronics. Still we managed to be homologated on the first day of the competition and participate to all the matches. Out of the 153 registered teams, we ended up on the 56th place out of the 112 teams that were homologated.

Us waiting for the beginning of our first match against team "Mars Attacks!"

Internship at CATIE Robotics​

I'm currently doing an internship at CATIE Robotics, the robotics department of CATIE, a technology transfer center in France. CATIE Robotics's participation to the Robocup has always came out with great results (RoboCup @Home German Open 2019 (2nd place), RoboCup @Home Sidney 2019 (3rd place), SciRoc Challenge (2nd place), (virtual) RoboCup @Home 2021 (2nd place) and RoboCup @Home Bordeaux (3rd place)). But their last participation in 2023 led them to the conclusion that they needed to build a new robot from scratch in order to compensate the TIAGo's robot's limitations.

The previous and beloved Epock robot based on the TIAGo robot from PAL Robotics (Image from robotics.catie.fr)

My internship is focused on the development of this new robot which is a 3-wheeled holonomic robot with a 7-DOF arm on top of it. Here are some of the things I have done so far (I'm halfway through my internship):

• Software Architecture: I have been working on the software architecture of the new robot, which is based on ROS2. The goal is to make the robot modular and easy to extend in the future by leveraging Docker containers and a microservices architecture with ROS2.

• Navigation: I have implemented and tuned the navigation stack of the robot, which is based on ROS2 Navigation 2. The robot uses a MPPI controller (Model Predictive Path Integral Controller) for its navigation, which allows it to navigate in complex environments with dynamic obstacles. Since the robot is for social robotics, I also added dynamic speed scaling for the robot to adapt its speed to its environment (See proxemics concept).

• Sensor Integration And Fusion: I have integrated and fused the data from the robot's sensors, which at the moment include 3 2D LiDARS, an optical odometry sensor and multiple RGB-D cameras. Our approach is to have cheaper sensors all around the robot to have a complete view of the environment and then to have more preicise sensors in the front of the robot to detect more detailed information for interaction with humans or object grabbing.

• Robot perception: I have developed CaSINO (CATIE System for Inference Node Orchestration), an AI orchestration system that allows the robot to run multiple AI models in parallel in ROS2. CaSINO is designed to be modular and easy to extend, allowing the robot to run different AI models for different tasks. It is based on ROS2 action servers and lifecycles, which allows it to be easily integrated into the robot's software architecture. It is also designed to be reliable and fault-tolerant : if one of the AI models crashes the system will continue to run the other models without interruption and restart the crashed model when possible.

• Embedded Software: The holonomic base of the robot is controlled through a micro-ROS node running on a custom embedded board running Zephyr RTOS. I have done some parts of the embedded software such as optical sensor integration and security features.

• Robot Design: I have evaluated the different sensors and computational units that will be used on the robot. The goal was to find the best trade-off between performance, power consumption, and cost.

I might post here about the progress of the robot at the end of my internship.

What I'm currently working on​

Working on AI orchestration during my internship undirectly led me to the rabithole of Reinforcement Learning for robotics. I might share some posts about the usage of Gymnasium and Stable Baselines3 for robotics in the future. This is a topic that I find very interesting as it allows robots to have more complex behaviors that could sometimes be hard to achieve with classical control methods. Obviously the downside is that those methods require a bit of time to train as well as an accurate simulation environment (especially when training for behaviors that requires the control of actuators).

On the other hand, I have also stared to work on a 3-wheeled holonomic robot for Eurobot 2026. The rules of the competition are not out yet but I want to work on the control of the robot in order to implement real-time curved trajectory calculation and execution on a microcontroller. I will probably post about this project in the future as well.