Evaluate your app’s performance before deploying it to production.

The phospho testing framework allows you to test your app with historical data, custom datasets, and custom tests.

The phospho python module parallelizes the function calls to speed up the testing process.

​
Getting started

To get started, install the phospho python module.

pip install -U phospho

Create a new file phospho_testing.py:

phospho_testing.py
import phospho

phospho_test = phospho.PhosphoTest()

In this file, you can then write your tests.

​
Backtesting

To use data from the phospho platform, you can use the backtest source loader.

phospho_testing.py
import phospho 

phospho_test = phospho.PhosphoTest()

@phospho_test.test(
    source_loader="backtest",  # Load data from logged phospho data
    source_loader_params={"sample_size": 3},
)
def test_backtest(message: phospho.lab.Message) -> str | None:
    client = phospho.lab.get_sync_client("mistral")
    response = client.chat.completions.create(
        model="mistral-small",
        messages=[
            {"role": "system", "content": "You are an helpful assistant"},
            {"role": message.role, "content": message.content},
        ],
    )
    return response.choices[0].message.content

​
Dataset .CSV, .XLSX, .JSON

To test with a custom dataset, you can use the dataset source loader.

phospho_testing.py
import phospho 

phospho_test = phospho.PhosphoTest()

@phospho_test.test(
    source_loader="dataset", 
    source_loader_params={"path": "path/to/dataset.csv"},
)
def test_backtest(column_a: str, column_b: str) -> str | None:
    client = phospho.lab.get_sync_client("mistral")
    response = client.chat.completions.create(
        model="mistral-small",
        messages=[
            {"role": "system", "content": "You are an helpful assistant"},
            {"role": "user", "content": column_a},
        ],
    )
    return response.choices[0].message.content

Supported file formats: csv, xlsx, json

The columns of the dataset file should match the function arguments.

Example of a local csv file:

path/to/dataset.csv
column_a, column_b
"What's larger, 3.9 or 3.11 ?", "3.11"

​
Custom tests

To write custom tests, you can just create a function and decorate it with @phospho_test.test().

At the end, add phospho.log to send the data to phospho for analysis.

phospho_testing.py
import phospho

phospho_test = phospho.PhosphoTest()

@phospho_test.test()
def test_simple():
    client = phospho.lab.get_sync_client("mistral")
    response = client.chat.completions.create(
        model="mistral-small",
        messages=[
            {"role": "system", "content": "You are an helpful assistant"},
            {"role": "user", "content": "What's bigger: 3.11 or 3.9 ?"},
        ],
    )
    response_text = response.choices[0].message.content
    # Use phospho.log to send the data to phospho for analysis
    phospho.log(
        input="What's bigger: 3.11 or 3.9 ?",
        output=response_text,
        # Specify the version_id of the test
        version_id=phospho_test.version_id,
    )

​
Run using python

To run the tests, use the run method of the PhosphoTest class.

phospho_testing.py
phospho_test.run()

The executor_type can be either:

  • parallel (default): parallelizes the backtest and dataset source loader calls.
  • parallel_jobs: all functions are called in parallel.
  • sequential: great for debugging.

​
Run using the phospho CLI

You can also use the phospho command line interface to run the tests. In the folder where phospho_testing.py is located, run:

phospho init # Run this only once
phospho test

The executor type can be specified with the --executor-type flag.

phospho test --executor-type=parallel_jobs

Learn more using the --help flag:

phospho test --help

phospo CLI

Learn how to install phospho command line interface

Was this page helpful?

Using a custom LLM providerLog to phospho with Javascript