Train robotics AI models

The phospho starter pack makes it easy to train robotics AI models by integrating with LeRobot from Hugging Face.

In this guide, we’ll show you how to train the ACT (Action Chunking Transformer) model using the phospho starter pack and LeRobot by Hugging Face.

​

What is LeRobot?

LeRobot is a platform designed to make real-world robotics more accessible for everyone. It provides pre-trained models, datasets, and tools in PyTorch.

It focuses on state-of-the-art approaches in imitation learning and reinforcement learning.

With LeRobot, you get access to:

• Pretrained models for robotics applications
• Human-collected demonstration datasets
• Simulated environments to test and refine AI models

Useful links:

• LeRobot on GitHub
• LeRobot on Hugging Face
• AI models for robotics

​

Step by step guide

In this guide, we will use the phospho starter pack to record a dataset and upload it to Hugging Face.

​

Prerequisites

1. You need an assembled SO-100 robot arm and cameras. Get the phosphot starter pack here.
2. Install the phosphobot software

curl -fsSL https://raw.githubusercontent.com/phospho-app/phosphobot/main/install.sh | bash

3. Connect your cameras to the computer. Start the phosphobot server.

phosphobot run

4. Complete the quickstart and check that you can control your robot.
5. You have the phosphobot teleoperation app is installed on your Meta Quest 2, Pro, 3 or 3s
6. You have a device to train your model. We recommend using a GPU for faster training.

​

1. Set up your Hugging Face token

To sync datasets, you need a Hugging Face token with write access. Follow these steps to generate one:

1. Log in to your Hugging Face account. You can create one here for free

2. Go to Profile and click Access Tokens in the sidebar.

3. Select the Write option to grant write access to your account. This is necessary for creating new datasets and uploading files. Name your token and click Create token.

4. Copy the token and save it in a secure place. You will need it later.

5. Make sure the phosphobot server is running. Open a browser and access localhost or phosphobot.local if you’re using the control module. Then go to the Admin Configuration.

6. Paste the Hugging Face token, and save it.

​

2. Set your dataset name and parameters

Go to the Admin Configuration page of your phospshobot dashboard. You can adjust settings. The most important are:

• Dataset Name: The name of the dataset you want to record.
• Task: A text description of the task you’re about to record. For example: “Pick up the lego brick and put it in the box”. This helps you remember what you recorded and is used by some AI models to understand the task.
• Camera: The cameras you want to record. By default, all cameras are recorded. You can select the cameras to record in the Admin Configuration.
• Video Codec: The video codec used to record the videos. The default is AVC1, which is the most efficient codec. If you’re having compatibility issues due to unavailable codecs (eg on Linux), switch to mp4v which is more compatible.

​

3. Control the robot in the Meta Quest app

The easiest way to record a dataset is to use the Meta Quest app.

1. In the Meta Quest, open the phospho teleop application. Wait a moment, then you should see a row displaying phosphobot or your computer name. Click the Connect button using the Trigger Button.

• Make sure you’re connected to the same WiFi as the phosphobot server.
• If you don’t see the server, check the IP address of the server in the phosphobot dashboard and enter it manually.

2. After connecting, you’ll see the list of connected cameras and recording options.

• Move the windows with the Grip button to organize your space.
• Enable preview to see the camera feed. Check the camera angles and adjust their positions if needed.

3. Press A once to start teleoperation and begin moving your controller.

   • The robot will naturally follow the movement of your controller. Press the Trigger button to close the gripper.
   • Press A again to stop the teleoperation. The robot will stop.

4. Press B to start recording. You can leave the default settings for your first attempt.

   • Press B again to stop the recording.
   • Press Y (left controller) to discard the recording.

5. Continue teleoperating and stop the recording by pressing B when you’re done.

6. The recording is automatically saved in LeRobot v2 format and uploaded to your HuggingFace account.

Go to your Hugging Face profile to see the uploaded datasets.

You can view it using the LeRobot Dataset Visualizer.

​

4. Train your first model

​

Train GR00T-N1-2B in one click with phosphobot cloud

To train a model, you can use the phosphobot cloud. This is the quickest way to train a model.

1. Enter the name of your dataset on Hugging Face (example: PLB/simple-lego-pickup-mono-2) in the AI Training and Control section.
2. Click on Train AI Model. Your model starts training. Training can take up to 3 hours. Follow the training using the button View trained models. Your model is uploaded to HuggingFace on the phospho-app account.
3. To control your robot with the trained model, go to the Control your robot section and enter the name of your model.

​

Train an ACT model on Replicate

Training ACT on your own machine can be hard. Video codecs, GPU acceleration, training time, and other factors can make it hard to train the model locally.

To help you, we provide a training script that you can run on the Replicate platform, a cloud service that provides GPU instances and scripts to train your AI models and run inference.

You’ll need to provide a Hugging Face dataset ID and token on which to train the policy.

​

Train an ACT model locally with LeRobot

This guide will show you how to train the ACT model locally using LeRobot for your SO-100 robot.

1. Install uv, the modern Python package manager.

# On macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

2. Set up training environment.

mkdir my_model
cd my_model
uv init
uv add phosphobot git+https://github.com/phospho-app/lerobot
git clone https://github.com/phospho-app/lerobot

3. (MacOS only) Set environment variables for torch compatibility:

export DYLD_LIBRARY_PATH="/opt/homebrew/lib:/usr/local/lib:$DYLD_LIBRARY_PATH"

4. (Optional) Add the Weight & Biases integration for training metrics tracking:

wandb login

5. Run training script - Adjust parameters based on your hardware:

uv run lerobot/lerobot/scripts/train.py \
 --dataset.repo_id=LegrandFrederic/Orange-brick-in-black-box \ # Replace with <HF_USERNAME>/<DATASET_NAME>
 --policy.type=act \ # Choose from act, diffusion, tdmpc, or vqbet
 --output_dir=outputs/train/phoshobot_test \
 --job_name=phosphobot_test \
 --policy.device=mps \  # Use 'cuda' for NVIDIA GPUs or 'cpu' if no GPU
 --wandb.enable=true # Optional

Trained models will be saved in lerobot/outputs/train/.

6. (Optional) Upload the model to Hugging Face. Login to HuggingFace CLI:

huggingface-cli login
# Enter your write token from https://huggingface.co/settings/tokens

HuggingFace model hub is a wrapper of Github LFS. Push the model to Hugging Face:

# From your training output directory
cd lerobot/outputs/train/phosphobot_test

# Initialize and push to Hub (replace <your-username> and <model-name>)
huggingface-cli repo-create <your-username>/<model-name> --type model
git lfs install
git add .
git commit -m "Add trained ACT model"
git push

​

5. Control your robot with the ACT model

1. Launch ACT inference server (Run on GPU machine):

# Download inference server script
curl -o server.py https://raw.githubusercontent.com/phospho-app/phosphobot/main/inference/ACT/server.py

# Start server
uv run server.py --model_id LegrandFrederic/Orange-brick-in-black-box # Replace with <YOUR_HF_MODEL_ID>

2. Make sure the phosphobot server is running to control your robot:

# Install it this way
curl -fsSL https://raw.githubusercontent.com/phospho-app/phosphobot/main/install.sh | bash
# Start it this way
phosphobot run

3. Create inference client script (Copy the content into my_model/client.py):

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "phosphobot",
# ]
#
# ///
from phosphobot.camera import AllCameras
from phosphobot.api.client import PhosphoApi
from phosphobot.am import ACT

import time
import numpy as np

# Connect to the phosphobot server
client = PhosphoApi(base_url="http://localhost:80")

# Get a camera frame
allcameras = AllCameras()

# Need to wait for the cameras to initialize
time.sleep(1)

# Instantiate the model
model = ACT() # Change server_url and server_port if needed

# Get the frames from the cameras
# We will use this model: LegrandFrederic/Orange-brick-in-black-box
# It requires 3 cameras as you can see in the config.json
# https://huggingface.co/LegrandFrederic/Orange-brick-in-black-box/blob/main/config.json

while True:
    images = [
        allcameras.get_rgb_frame(camera_id=0, resize=(240, 320)),
        allcameras.get_rgb_frame(camera_id=1, resize=(240, 320)),
        allcameras.get_rgb_frame(camera_id=2, resize=(240, 320)),
    ]

    # Get the robot state
    state = client.control.read_joints()

    inputs = {"state": np.array(state.angles_rad), "images": np.array(images)}

    # Go through the model
    actions = model(inputs)

    for action in actions:
        # Send the new joint postion to the robot
        client.control.write_joints(angles=action.tolist())
        # Wait to respect frequency control (30 Hz)
        time.sleep(1 / 30)

4. Run the inference script:

uv run client.py

Stop the script by pressing Ctrl + C.

​

What’s next?

Next, you can use the trained model to control your robot. Head to our guide to get started!