Quickstart

Here is a complete, runnable script — save it as quickstart.py and run python quickstart.py. The rest of the page explains each part.

# quickstart.py
import random
from entropy import Suite, Dataset, Case, assert_stable

# 1. The agent under test: any callable. Here a non-deterministic one.
def my_agent(inp):
    r = random.random()
    if r < 0.7:
        return "A"
    if r < 0.9:
        return "B"
    return "C"

# 2. A dataset: inputs plus the expected (or check) outputs.
dataset = Dataset([
    Case(input="q1", expected="A"),
    Case(input="q2", expected="B"),
])

# 3. Run the Monte Carlo suite: `trials` runs per case.
results = Suite(seed=42).run(my_agent, dataset, trials=100)
print(results)   # 50+ metrics: success_rate, behavioral_entropy, ...

# 4. Regression gate: fail the run if behavior drifts out of bounds.
assert_stable(
    results,
    success_rate__gt=0.3,
    behavioral_entropy__gt=0.0,   # non-deterministic -> non-zero entropy
    drift_score__lt=0.2,
)
print("assert_stable: OK")

EntroPy evaluates an agent by running it many times and measuring the distribution of its behavior. This takes three objects:

1. Define an agent

An agent is any callable. It may return a plain value, an AgentRun, or a dict with output/events/cost.

import random

def my_agent(inp):
    r = random.random()
    if r < 0.7:
        return "A"
    if r < 0.9:
        return "B"
    return "C"

2. Build a dataset

from entropy import Dataset, Case

dataset = Dataset([
    Case(input="q1", expected="A"),
    Case(input="q2", expected="B"),
])

3. Run the suite

from entropy import Suite

results = Suite(seed=42).run(my_agent, dataset, trials=100)
print(results)

This prints the canonical metric set — success_rate, behavioral_entropy, drift_score, recovery_score, and more (50+ metrics total).

Reproducibility

Always pass a fixed seed to Suite so evaluations are reproducible and comparable across runs and CI.

4. Assert stability (regression gate)

Use assert_stable as a guard in tests or CI:

from entropy import assert_stable

assert_stable(
    results,
    success_rate__gt=0.3,
    behavioral_entropy__gt=0.0,   # non-deterministic -> non-zero entropy
    drift_score__lt=0.2,
)

Threshold operators are __gt, __gte, __lt, __lte, or exact equality when omitted.

What's next