Train robotics AI models

​

What is LeRobot?

• Pretrained models for robotics applications
• Human-collected demonstration datasets
• Simulated environments to test and refine AI models

• LeRobot on GitHub
• LeRobot on Hugging Face
• AI models for robotics

​

Step by step guide

​

Prerequisites

1. You need an assembled SO-100 robot arm and cameras. Get the phosphot starter pack here.
2. Install the phosphobot software

curl -fsSL https://raw.githubusercontent.com/phospho-app/phosphobot/main/install.sh | bash

3. Connect your cameras to the computer. Start the phosphobot server.

phosphobot run

4. Complete the quickstart and check that you can control your robot.
5. You have the phosphobot teleoperation app is installed on your Meta Quest 2, Pro, 3 or 3s

6. You have a device to train your model. We recommend using a GPU for faster training.

​

1. Set up your Hugging Face token

1. Log in to your Hugging Face account. You can create one here for free
2. Go to Profile and click Access Tokens in the sidebar.
3. Select the Write option to grant write access to your account. This is necessary for creating new datasets and uploading files. Name your token and click Create token.
4. Copy the token and save it in a secure place. You will need it later.
5. Make sure the phosphobot server is running. Open a browser and access localhost or phosphobot.local if you’re using the control module. Then go to the Admin Configuration.
6. Paste the Hugging Face token, and save it.

​

2. Set your dataset name and parameters

• Dataset Name: The name of the dataset you want to record.
• Task: A text description of the task you’re about to record. For example: “Pick up the lego brick and put it in the box”. This helps you remember what you recorded and is used by some AI models to understand the task.
• Camera: The cameras you want to record. By default, all cameras are recorded. You can select the cameras to record in the Admin Configuration.
• Video Codec: The video codec used to record the videos. The default is AVC1, which is the most efficient codec. If you’re having compatibility issues due to unavailable codecs (eg on Linux), switch to mp4v which is more compatible.

​

3. Control the robot in the Meta Quest app

1. In the Meta Quest, open the phospho teleop application. Wait a moment, then you should see a row displaying phosphobot or your computer name. Click the Connect button using the Trigger Button.

2. After connecting, you’ll see the list of connected cameras and recording options.

• Move the windows with the Grip button to organize your space.
• Enable preview to see the camera feed. Check the camera angles and adjust their positions if needed.

3. Press A once to start teleoperation and begin moving your controller.
   • The robot will naturally follow the movement of your controller. Press the Trigger button to close the gripper.
   • Press A again to stop the teleoperation. The robot will stop.
4. Press B to start recording. You can leave the default settings for your first attempt.
   • Press B again to stop the recording.
   • Press Y (left controller) to discard the recording.
5. Continue teleoperating and stop the recording by pressing B when you’re done.
6. The recording is automatically saved in LeRobot v2 format and uploaded to your HuggingFace account.

​

4. Train your first model

​

Train GR00T-N1-2B in one click with phosphobot cloud

1. Enter the name of your dataset on Hugging Face (example: PLB/simple-lego-pickup-mono-2) in the AI Training and Control section.
2. Click on Train AI Model. Your model starts training. Training can take up to 3 hours. Follow the training using the button View trained models. Your model is uploaded to HuggingFace on the phospho-app account.
3. To control your robot with the trained model, go to the Control your robot section and enter the name of your model.

​

Train an ACT model locally with LeRobot

1. Install uv, the modern Python package manager.

# On macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

2. Set up training environment.

mkdir my_model
cd my_model
uv init
uv add phosphobot git+https://github.com/phospho-app/lerobot
git clone https://github.com/phospho-app/lerobot

3. (MacOS only) Set environment variables for torch compatibility:

export DYLD_LIBRARY_PATH="/opt/homebrew/lib:/usr/local/lib:$DYLD_LIBRARY_PATH"

4. (Optional) Add the Weight & Biases integration for training metrics tracking:

wandb login

5. Run training script - Adjust parameters based on your hardware:

uv run lerobot/lerobot/scripts/train.py \
 --dataset.repo_id=LegrandFrederic/Orange-brick-in-black-box \ # Replace with <HF_USERNAME>/<DATASET_NAME>
 --policy.type=act \ # Choose from act, diffusion, tdmpc, or vqbet
 --output_dir=outputs/train/phoshobot_test \
 --job_name=phosphobot_test \
 --policy.device=mps \  # Use 'cuda' for NVIDIA GPUs or 'cpu' if no GPU
 --wandb.enable=true # Optional

6. (Optional) Upload the model to Hugging Face. Login to HuggingFace CLI:

huggingface-cli login
# Enter your write token from https://huggingface.co/settings/tokens

# From your training output directory
cd lerobot/outputs/train/phosphobot_test

# Initialize and push to Hub (replace <your-username> and <model-name>)
huggingface-cli repo-create <your-username>/<model-name> --type model
git lfs install
git add .
git commit -m "Add trained ACT model"
git push

​

5. Control your robot with the ACT model

1. Launch ACT inference server (Run on GPU machine):

# Download inference server script
curl -o server.py https://raw.githubusercontent.com/phospho-app/phosphobot/main/inference/ACT/server.py

# Start server
uv run server.py --model_id LegrandFrederic/Orange-brick-in-black-box # Replace with <YOUR_HF_MODEL_ID>

2. Make sure the phosphobot server is running to control your robot:

# Install it this way
curl -fsSL https://raw.githubusercontent.com/phospho-app/phosphobot/main/install.sh | bash
# Start it this way
phosphobot run

3. Create inference client script (Copy the content into my_model/client.py):

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "phosphobot",
# ]
#
# ///
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "phosphobot",
# ]
#
# ///

from phosphobot.camera import AllCameras
import httpx
from phosphobot.am import ACT
import time
import numpy as np

# Initialize hardware interfaces
PHOSPHOBOT_API_URL = "http://localhost:80"
allcameras = AllCameras()
time.sleep(1)  # Camera warmup

# Connect to ACT server
model = ACT()

while True:
    # Capture multi-camera frames (adjust camera IDs and size as needed)
    images = [allcameras.get_rgb_frame(0, resize=(240, 320))]

    # Get current robot state
    state = httpx.post(f"{PHOSPHOBOT_API_URL}/joints/read").json()

    # Generate actions
    actions = model(
        {"state": np.array(state["angles_rad"]), "images": np.array(images)}
    )

    # Execute actions at 30Hz
    for action in actions:
        httpx.post(
            f"{PHOSPHOBOT_API_URL}/joints/write", json={"angles": action.tolist()}
        )
        time.sleep(1 / 30)

4. Run the inference script:

uv run client.py

​

What’s next?