Remote Rollout Processor Hello World

Run distributed LLM evaluations with remote rollout processing and automatic trace collection. No manual trace management required.

📖 New to Remote Rollout Processing? Read the complete Remote Rollout Processor Tutorial first - this repository serves as the working example for that tutorial.

Quick Start

Installation

pip install eval-protocol

Environment Setup

Set up your Fireworks API Key for model inference:

export FIREWORKS_API_KEY="your_fireworks_key"

Running the Example

Option 1: Python Server

Start the Python server:

python -m remote_server

In another terminal, run the evaluation test:

pytest quickstart.py -vs

Option 2: TypeScript Server

Start the TypeScript server:

cd typescript-server
pnpm install
pnpm run dev

In another terminal, run the same evaluation test:

pytest quickstart.py -vs

What's Happening

1. /init triggers one rollout: Eval Protocol makes a POST /init request with the row payload and correlation metadata to our server on http://127.0.0.1:3000, which triggers the rollout (in this case a simple chat completion asking "What is the capital of France").
2. Send logs via FireworksTracingHttpHandler: Our server emits structured logs tagged with the rollout’s correlation fields.
3. Send chat completions and store as trace: Our chat completion calls are recorded as traces in Fireworks.
4. Once rollout finished, pull full trace and evaluate: Eval Protocol polls Fireworks for a completion signal, then pulls the trace back and scores.

Viewing Results

After running the test, start the local UI server:

ep logs

At http://localhost:8000, you'll see results like:

Learn More

• Eval Protocol Documentation
• Remote Rollout Processor Tutorial
• Fireworks AI Platform