phospho starter pack documentation

You just trained an AI model and now you want to use it to control your robot. Let’s see how you can do that.

Disclaimer: Letting an AI control your robot carries risk. Clear the area from pets, people and objects. You are the only one responsible for any damage caused to your robot or its surroundings.

Control your robot from the phosphobot dashboard

If you finetuned a GR00T-N1-2B model and pushed it to huggingface, you can use it directly from the phosphobot dashboard.

You can fine tune the model in a single click from the dashboard. Go here to learn how.

Connect your robots and your cameras to your computer. Run the phosphobot server and go to the phosphobot dashboard in your browser: http://localhost

phosphobot run

Create a phospho account or log in by clicking on the Sign in button in the top right corner.
(If not already done) Add your Hugging Face token in the Admin Settings tab with Write authorization. Read the full guide here.
In the AI Training and Control section, enter the instruction you want to give the robot and click on Go to AI Control. Accept the disclaimer. You’ll be redirected to the AI Control page.

In the Model ID, enter the name of your model on Hugging Face (example: phospho-app/YOUR_DATASET_NAME-A_RANDOM_ID). Double check the camera angles so that they match the ones you used to record the dataset.
Click on Start AI Control. Please wait: the first time, starting a GPU instance and loading the model can take up to 60 seconds. Then the robot will start moving.

You can pause, resume, and stop the AI control at any time by clicking on the control buttons.

You can edit the Instruction field to change the instruction and run it again to see how the robot reacts.

Discord

Join the Discord to ask questions and share your demos!

How to control your robot in Python?

If you’re using a different model or want more fine-grained control, you can use the phosphobot python module to control your robot with an AI model.

1. Setup an inference server

First, you need to setup an inference server. This server runs on a beefy machine with a GPU that can run the AI model. It can be your own machine, a cloud server, or a dedicated server.

If you choose a remote location, chose the closest location to minimize latency.

To setup the inference server, follow the instructions in the link below:

Setup the inference server

How to setup the inference server?

2. Call your inference server from a python script

Open a terminal and run the phosphobot server.

phosphobot run

Then, create a new python file called inference.py. Inside, copy the content of an example script below.

Example script for ACT

# pip install --upgrade phosphobot
from phosphobot.camera import AllCameras
from phosphobot.api.client import PhosphoApi
from phosphobot.am import ACT

import time
import numpy as np

# Connect to the phosphobot server
client = PhosphoApi(base_url="http://localhost:80")

# Get a camera frame
allcameras = AllCameras()

# Need to wait for the cameras to initialize
time.sleep(1)

# Instantiate the model
model = ACT()

# Get the frames from the cameras
# We will use this model: LegrandFrederic/Orange-brick-in-black-box
# It requires 3 cameras as you can see in the config.json
# https://huggingface.co/LegrandFrederic/Orange-brick-in-black-box/blob/main/config.json

while True:
    images = [
        allcameras.get_rgb_frame(camera_id=0, resize=(240, 320)),
        allcameras.get_rgb_frame(camera_id=1, resize=(240, 320)),
        allcameras.get_rgb_frame(camera_id=2, resize=(240, 320)),
    ]

    # Get the robot state
    state = client.control.read_joints()

    inputs = {"state": np.array(state.angles_rad), "images": np.array(images)}

    # Go through the model
    actions = model(inputs)

    for action in actions:
        # Send the new joint postion to the robot
        client.control.write_joints(angles=action.tolist())
        # Wait to respect frequency control (30 Hz)
        time.sleep(1 / 30)

Example script for Pi0

#pip install --upgrade phosphobot
from phosphobot.camera import AllCameras
from phosphobot.api.client import PhosphoApi
from phosphobot.am import Pi0

import time
import numpy as np

# Connect to the phosphobot server
client = PhosphoApi(base_url="http://localhost:80")

# Get a camera frame
allcameras = AllCameras()

# Need to wait for the cameras to initialize
time.sleep(1)

# Instantiate the model
model = Pi0(server_url="YOUR_SERVER_URL")

while True:
    # Get the frames from the cameras
    # We will use this model: PLB/pi0-so100-orangelegobrick-wristcam
    # It requires 2 cameras (a context cam and a wrist cam)
    images = [
        allcameras.get_rgb_frame(camera_id=0, resize=(240, 320)),
        allcameras.get_rgb_frame(camera_id=1, resize=(240, 320)),
    ]

    # Get the robot state
    state = client.control.read_joints()

    inputs = {
        "state": np.array(state.angles_rad),
        "images": np.array(images),
        "prompt": "Pick up the orange brick",
    }

    # Go through the model
    actions = model(inputs)

    for action in actions:
        # Send the new joint postion to the robot
        client.control.write_joints(angles=action.tolist())
        # Wait to respect frequency control (30 Hz)
        time.sleep(1 / 30)

Example script for gr00t-n1

You need to install the torch and zmq libraries.

pip install torch zmq

You also need to run a GPU with the gr00t model and inference server. Use this repo to run the server.

# pip install --upgrade phosphobot
from phosphobot.camera import AllCameras
from phosphobot.api.client import PhosphoApi
from phosphobot.am import Gr00tN1

import time
import numpy as np

# Connect to the phosphobot server
client = PhosphoApi(base_url="http://localhost:80")

# Get a camera frame
allcameras = AllCameras()

# Need to wait for the cameras to initialize
time.sleep(1)

# Instantiate the model
model = Gr00tN1(server_url="YOUR_SERVER_URL")

while True:
    images = [
        allcameras.get_rgb_frame(camera_id=0, resize=(240, 320)),
        allcameras.get_rgb_frame(camera_id=1, resize=(240, 320)),
    ]

    # Get the robot state
    state = client.control.read_joints()

    inputs = {
        "state": np.array(state.angles_rad),
        "images": np.array(images),
        "prompt": "Pick up the orange brick",
    }

    # Go through the model
    actions = model(inputs)

    for action in actions:
        # Send the new joint postion to the robot
        client.control.write_joints(angles=action.tolist())
        # Wait to respect frequency control (30 Hz)
        time.sleep(1 / 30)

To run the script, install the phosphobot python module. Then, run the script.

pip install phosphobot
python your_script.py

What’s next?

Discord

Join the Discord to ask questions, get help from others and get updates (we ship almost daily)

Policies

Learn more about Robotics AI models

Getting Started

Basic Usage

Learn about robotics

Hardware

Examples

API Reference

Control robot with AI models

Control your robot from the phosphobot dashboard

Discord

How to control your robot in Python?

1. Setup an inference server

Setup the inference server

2. Call your inference server from a python script

Example script for ACT

Example script for Pi0

Example script for gr00t-n1

Other models

What’s next?

Discord

Policies

Getting Started

Basic Usage

Learn about robotics

Hardware

Examples

API Reference

​Control your robot from the phosphobot dashboard

Discord

​How to control your robot in Python?

​1. Setup an inference server

Setup the inference server

​2. Call your inference server from a python script

​What’s next?

Discord

Policies

Control your robot from the phosphobot dashboard

How to control your robot in Python?

1. Setup an inference server

2. Call your inference server from a python script

What’s next?