Evaluate, diagnose and compare AI agents

Use EntroPy to inspect repeated agent failures, estimate success within several attempts, and compare prompt or model versions on the same test cases. These methods run locally and require no additional dependencies.

Run a complete evaluation and comparison

Install with pip install entropy-ai. Save the following as evaluation.py and run it with Python. This example uses no provider API.

from entropy import evaluate_detailed, compare_agents

cases = {" Hello ": "hello", "WORLD": "world", " Test ": "test"}

def baseline(text):
    return text.strip()

def candidate(text):
    return text.strip().lower()

report = evaluate_detailed(candidate, cases, trials=10, pass_k=[1, 5, 10])
print(report["pass_at_k"])
print(report["cases"][0]["confidence_interval"])

comparison = compare_agents(
    {"baseline": baseline, "candidate": candidate}, cases,
    trials=10, resamples=1000, seed=42,
)
print(comparison["comparisons"]["candidate"]["status"])

Inspect failures per input

evaluate_detailed returns summary (the standard metric dictionary), total_trials, macro-averaged pass_at_k, confidence and cases. Each case contains its input, metadata, trial and success counts, success rate, Wilson interval, behavioral entropy, mean cost, mean latency and failed outputs.

Failure entries use 1-based trial numbers and retain output, the agent's error metadata, and recorded error-event messages in errors. A wrong answer can be a failure without an exception. Cost and latency come from the agent's recorded data; missing latency is None. No detailed report is automatically written to disk.

Provide expected answers or check= to measure correctness. Otherwise success means no reported error. Failed outputs and inputs can contain sensitive data; inspect reports before sharing them.

Estimate success within k attempts

pass_at_k(total, correct, k) estimates the chance of at least one successful result in k attempts using 1 - C(n-c, k) / C(n, k). For 2 successes among 10 trials, pass_at_k(10, 2, 3) is approximately 0.5333. Each k must be positive and no larger than the number of trials.

In detailed reports, each input gets a separate estimate and the top-level value averages inputs equally. Attempts must use the same success criterion and be independent samples. This estimate does not model a stateful retry policy that learns from previous failures.

Report uncertainty

wilson_interval(correct, total, confidence=0.95) returns lower and upper bounds for a binomial success proportion. Ten successful trials give approximately [0.7225, 1.0], rather than claiming perfect certainty. No observations return [0.0, 1.0].

The standard confidence_interval metric now uses Wilson bounds over all observed trials. Pooling heterogeneous tasks is an approximation under independent-trial assumptions. Detailed reports also give per-input intervals. Their confidence= option applies to both the per-input and summary Wilson intervals when that metric is selected.

bootstrap_interval(values, confidence=0.95, resamples=2000, seed=0) estimates a percentile interval for the mean of at least two finite numbers. It resamples observations with replacement. Constant data produces a point interval; bootstrap resampling cannot reveal unobserved variation.

Compare prompts, models or agent versions

compare_agents accepts a mapping of at least two names to synchronous callables. The first name is the baseline unless you pass baseline=. Every agent runs all inputs for the requested number of trials. Generators are materialized once so every candidate receives the same cases. Agents should not mutate shared inputs in place.

The result includes detailed agents reports and comparisons against the baseline. Each comparison contains metric deltas (candidate minus baseline), counts of improved and regressed inputs, and a bootstrap success_rate_interval. Resampling pairs per-input success-rate differences; it does not pair unrelated trial outputs.

These are exploratory comparisons without multiple-comparison correction. More representative inputs and independent trials improve the evidence. A seeded bootstrap is reproducible; it does not seed a remote model. Cost and latency deltas do not determine the success-rate status. A latency delta is None if either agent lacks latency evidence for any input; comparison reports never substitute cost for elapsed time.

Use async agents

await evaluate_detailed_async(agent, inputs, trials=20, pass_k=[1, 5]) returns the same report. Trials run sequentially. Checks stay synchronous and cancellation propagates. compare_agents currently accepts synchronous agents; use detailed async reports to inspect async agents individually.

Use the CLI

entropy run --agent mymodule:agent --dataset cases.json --trials 20 --detailed --pass-k 1 5 10 --out details.json
entropy benchmark --agents mymodule:old mymodule:new --dataset cases.json --trials 20 --paired --resamples 2000 --out comparison.json

For paired comparisons, --baseline mymodule:old selects the reference and --confidence 0.99 changes interval coverage. Without the new flags, existing run and benchmark output formats are preserved.

Method references

Next: choose evaluation metrics, record agent tool use, or browse the Python API.