API Reference
Everything below is importable from the top-level entropy
package (e.g. from entropy import Suite). Modules are listed
by area; see the Metrics Catalog for the full
metric list.
Core — entropy.core
| Symbol | Signature / Notes |
|---|---|
Suite(seed=0, metrics=None) | Monte Carlo runner. .run(agent, dataset, trials=100, extractor=None) -> dict. |
evaluate_async(agent, inputs, *, trials=100, seed=0, metrics=None, expected=None, check=None) | Await an async agent evaluation; returns the same metric dictionary as evaluate. |
await Suite().run_async(agent, dataset, trials=100, extractor=None, expected=None, check=None) | Sequential async trials, with recorded batches. Agent cancellation propagates. Checks and extractors are synchronous; existing framework adapters remain synchronous. |
assert_stable(results, **thresholds) | Regression gate; thresholds like success_rate__gt=0.9. Raises on violation. |
AgentRun | Dataclass: run_id, input, output, events, cost, metadata. |
Event / typed events | ActionEvent, ReasoningEvent, ToolCallEvent, ObservationEvent, MemoryReadEvent, MemoryWriteEvent, ErrorEvent, StateTransitionEvent. |
Trace | Dataclass: runs, nodes, edges, metadata + .json() .df() .graph() .otel() .zip(). |
Node, Edge | Behavior graph primitives. |
instrument(agent) | Wrap a callable (or framework agent) to return an AgentRun. |
trace(), event(type, name, **data), current_trace() | Manual tracing context & emitters. |
Detailed evaluation and statistical methods
All methods below are exported from entropy.
See the evaluation and comparison guide for examples and assumptions.
| Method | Purpose |
|---|---|
evaluate_detailed(agent, inputs, *, trials=100, seed=0, metrics=None, expected=None, check=None, pass_k=(1,), confidence=0.95) | Aggregate metrics, per-input evidence, Pass@k, uncertainty and failed outputs. |
await evaluate_detailed_async(...) | The same arguments and report for async callables. |
compare_agents(agents, inputs, *, trials=100, baseline=None, seed=0, expected=None, check=None, confidence=0.95, resamples=2000) | Paired per-input bootstrap comparisons against a named baseline. Synchronous agents only. |
pass_at_k(total, correct, k=1) | Estimate success in k attempts. Requires 1 <= k <= total. |
wilson_interval(correct, total, confidence=0.95) | Wilson score interval for a binomial success proportion. |
bootstrap_interval(values, *, confidence=0.95, resamples=2000, seed=0) | Percentile bootstrap interval for a mean of at least two finite numeric observations. |
Metrics — entropy.metrics
| Symbol | Notes |
|---|---|
metric(name=None) | Decorator registering a metric class. |
Metric | Base class; implement compute(self, batch). |
Batch | Per-input aggregation: outputs, behaviors, actions, tool_calls, successes, costs, errored, recovered. |
default_metrics() | List the canonical Suite metric names. |
layers | Mapping of evaluation layer → metrics (see Metrics). |
Datasets — entropy.datasets
| Symbol | Notes |
|---|---|
Dataset(list) | List of cases; from_json / from_jsonl / from_yaml / from_csv / from_hf, .save(). |
Case(input, expected=None, check=None, metadata=None) | Generic case. |
GoldenCase, Scenario, BehaviorCase, FailureCase, AdversarialCase | Specialized cases. |
Simulation — entropy.simulation
| Symbol | Notes |
|---|---|
ScriptedUserSimulator(script), RandomUserSimulator(pool, seed=0), LLMUserSimulator(client, model) | User turn generators with .next() / .reset(). |
StatefulEnv(state=None), GridWorld(w, h, goal) | Environments with .step(action) / .reset(). |
AdversarialSimulator(seed=0, attacks=None) | .iterate(text), .perturb(text, kind). |
Chaos — entropy.chaos
| Symbol | Notes |
|---|---|
ChaosRunner(seed=0) | .run(agent, dataset, faults, trials=50) -> dict, .summary(result). |
chaos_inject(agent, fault, p=1.0, seed=0) | Wrap an agent so a fault fires with probability p. |
chaos_available() | List available fault names. |
Observability — entropy.observability
| Symbol | Notes |
|---|---|
Watcher(agent, dataset, trials=20, seed=0) | .watch(rounds, interval, render), .series(), .anomalies(), .failure_clusters(). |
render_metric_heatmap(history, path, metrics=None) | Writes a PNG heatmap. |
Reports & Dashboard — entropy.reports, entropy.dashboard
| Symbol | Notes |
|---|---|
export_report(results, fmt, path, title="EntroPy Report") | HTML / JSON / Markdown / PDF / Jupyter. |
export(results, fmt, path, title) | Lower-level dispatcher. |
build_dashboard(results, out_dir), serve_dashboard(out_dir, port=8000) | Static dashboard HTML. |
Integrations — entropy.integrations
| Symbol | Notes |
|---|---|
from_langchain / from_langgraph / from_openai / from_crewai / from_pydanticai / from_autogen / from_google_adk / from_mcp / from_custom | Wrap a framework agent into a callable. |
adapter(name), Adapter, register_adapter | Define custom adapters. |
find_adapter(agent), list_adapters() | Discovery helpers. |
Plugins — entropy.plugins
| Symbol | Notes |
|---|---|
exporter(name=None) | Register a report exporter fn(results, path) -> str. |
discover(group="entropy.plugins") | Import entry-point plugins. |