Evaluate, diagnose and compare AI agents
Use EntroPy to inspect repeated agent failures, estimate success within several attempts, and compare prompt or model versions on the same test cases. These methods run locally and require no additional dependencies.
Run a complete evaluation and comparison
Install with pip install entropy-ai. Save the following as
evaluation.py and run it with Python. This example uses no provider API.
from entropy import evaluate_detailed, compare_agents
cases = {" Hello ": "hello", "WORLD": "world", " Test ": "test"}
def baseline(text):
return text.strip()
def candidate(text):
return text.strip().lower()
report = evaluate_detailed(candidate, cases, trials=10, pass_k=[1, 5, 10])
print(report["pass_at_k"])
print(report["cases"][0]["confidence_interval"])
comparison = compare_agents(
{"baseline": baseline, "candidate": candidate}, cases,
trials=10, resamples=1000, seed=42,
)
print(comparison["comparisons"]["candidate"]["status"])
Inspect failures per input
evaluate_detailed returns summary (the standard metric
dictionary), total_trials, macro-averaged pass_at_k,
confidence and cases. Each case contains its input,
metadata, trial and success counts, success rate, Wilson interval, behavioral
entropy, mean cost, mean latency and failed outputs.
Failure entries use 1-based trial numbers and retain output,
the agent's error metadata, and recorded error-event messages in
errors. A wrong answer can be a failure without an exception.
Cost and latency come from the agent's recorded data; missing latency is
None. No detailed report is automatically written to disk.
Provide expected answers or check= to measure correctness.
Otherwise success means no reported error. Failed outputs and inputs can
contain sensitive data; inspect reports before sharing them.
Estimate success within k attempts
pass_at_k(total, correct, k) estimates the chance of at least one
successful result in k attempts using 1 - C(n-c, k) / C(n, k).
For 2 successes among 10 trials, pass_at_k(10, 2, 3) is approximately
0.5333. Each k must be positive and no larger than the number of trials.
In detailed reports, each input gets a separate estimate and the top-level value averages inputs equally. Attempts must use the same success criterion and be independent samples. This estimate does not model a stateful retry policy that learns from previous failures.
Report uncertainty
wilson_interval(correct, total, confidence=0.95) returns lower and
upper bounds for a binomial success proportion. Ten successful trials give
approximately [0.7225, 1.0], rather than claiming perfect certainty.
No observations return [0.0, 1.0].
The standard confidence_interval metric now uses Wilson bounds
over all observed trials. Pooling heterogeneous tasks is an approximation
under independent-trial assumptions. Detailed reports also give per-input
intervals. Their confidence= option applies to both the per-input
and summary Wilson intervals when that metric is selected.
bootstrap_interval(values, confidence=0.95, resamples=2000, seed=0)
estimates a percentile interval for the mean of at least two finite numbers.
It resamples observations with replacement. Constant data produces a point
interval; bootstrap resampling cannot reveal unobserved variation.
Compare prompts, models or agent versions
compare_agents accepts a mapping of at least two names to synchronous
callables. The first name is the baseline unless you pass baseline=.
Every agent runs all inputs for the requested number of trials. Generators
are materialized once so every candidate receives the same cases. Agents should
not mutate shared inputs in place.
The result includes detailed agents reports and
comparisons against the baseline. Each comparison contains metric
deltas (candidate minus baseline), counts of improved and regressed inputs,
and a bootstrap success_rate_interval. Resampling pairs per-input
success-rate differences; it does not pair unrelated trial outputs.
improved: the entire observed bootstrap interval is positive.regressed: the entire interval is negative.inconclusive: the interval includes zero.insufficient_data: only one input; the interval isNone.
These are exploratory comparisons without multiple-comparison correction.
More representative inputs and independent trials improve the evidence.
A seeded bootstrap is reproducible; it does not seed a remote model.
Cost and latency deltas do not determine the success-rate status. A latency
delta is None if either agent lacks latency evidence for any input;
comparison reports never substitute cost for elapsed time.
Use async agents
await evaluate_detailed_async(agent, inputs, trials=20, pass_k=[1, 5])
returns the same report. Trials run sequentially. Checks stay synchronous and
cancellation propagates. compare_agents currently accepts synchronous
agents; use detailed async reports to inspect async agents individually.
Use the CLI
entropy run --agent mymodule:agent --dataset cases.json --trials 20 --detailed --pass-k 1 5 10 --out details.json
entropy benchmark --agents mymodule:old mymodule:new --dataset cases.json --trials 20 --paired --resamples 2000 --out comparison.json
For paired comparisons, --baseline mymodule:old selects the reference
and --confidence 0.99 changes interval coverage. Without the new
flags, existing run and benchmark output formats are preserved.
Method references
- Evaluating Large Language Models Trained on Code: Pass@k estimator.
- NIST confidence interval guidance: Wilson score intervals.
Next: choose evaluation metrics, record agent tool use, or browse the Python API.