API Reference

Everything below is importable from the top-level entropy package (e.g. from entropy import Suite). Modules are listed by area; see the Metrics Catalog for the full metric list.

Core — entropy.core

SymbolSignature / Notes
Suite(seed=0, metrics=None)Monte Carlo runner. .run(agent, dataset, trials=100, extractor=None) -> dict.
evaluate_async(agent, inputs, *, trials=100, seed=0, metrics=None, expected=None, check=None)Await an async agent evaluation; returns the same metric dictionary as evaluate.
await Suite().run_async(agent, dataset, trials=100, extractor=None, expected=None, check=None)Sequential async trials, with recorded batches. Agent cancellation propagates. Checks and extractors are synchronous; existing framework adapters remain synchronous.
assert_stable(results, **thresholds)Regression gate; thresholds like success_rate__gt=0.9. Raises on violation.
AgentRunDataclass: run_id, input, output, events, cost, metadata.
Event / typed eventsActionEvent, ReasoningEvent, ToolCallEvent, ObservationEvent, MemoryReadEvent, MemoryWriteEvent, ErrorEvent, StateTransitionEvent.
TraceDataclass: runs, nodes, edges, metadata + .json() .df() .graph() .otel() .zip().
Node, EdgeBehavior graph primitives.
instrument(agent)Wrap a callable (or framework agent) to return an AgentRun.
trace(), event(type, name, **data), current_trace()Manual tracing context & emitters.

Detailed evaluation and statistical methods

All methods below are exported from entropy. See the evaluation and comparison guide for examples and assumptions.

MethodPurpose
evaluate_detailed(agent, inputs, *, trials=100, seed=0, metrics=None, expected=None, check=None, pass_k=(1,), confidence=0.95)Aggregate metrics, per-input evidence, Pass@k, uncertainty and failed outputs.
await evaluate_detailed_async(...)The same arguments and report for async callables.
compare_agents(agents, inputs, *, trials=100, baseline=None, seed=0, expected=None, check=None, confidence=0.95, resamples=2000)Paired per-input bootstrap comparisons against a named baseline. Synchronous agents only.
pass_at_k(total, correct, k=1)Estimate success in k attempts. Requires 1 <= k <= total.
wilson_interval(correct, total, confidence=0.95)Wilson score interval for a binomial success proportion.
bootstrap_interval(values, *, confidence=0.95, resamples=2000, seed=0)Percentile bootstrap interval for a mean of at least two finite numeric observations.

Metrics — entropy.metrics

SymbolNotes
metric(name=None)Decorator registering a metric class.
MetricBase class; implement compute(self, batch).
BatchPer-input aggregation: outputs, behaviors, actions, tool_calls, successes, costs, errored, recovered.
default_metrics()List the canonical Suite metric names.
layersMapping of evaluation layer → metrics (see Metrics).

Datasets — entropy.datasets

SymbolNotes
Dataset(list)List of cases; from_json / from_jsonl / from_yaml / from_csv / from_hf, .save().
Case(input, expected=None, check=None, metadata=None)Generic case.
GoldenCase, Scenario, BehaviorCase, FailureCase, AdversarialCaseSpecialized cases.

Simulation — entropy.simulation

SymbolNotes
ScriptedUserSimulator(script), RandomUserSimulator(pool, seed=0), LLMUserSimulator(client, model)User turn generators with .next() / .reset().
StatefulEnv(state=None), GridWorld(w, h, goal)Environments with .step(action) / .reset().
AdversarialSimulator(seed=0, attacks=None).iterate(text), .perturb(text, kind).

Chaos — entropy.chaos

SymbolNotes
ChaosRunner(seed=0).run(agent, dataset, faults, trials=50) -> dict, .summary(result).
chaos_inject(agent, fault, p=1.0, seed=0)Wrap an agent so a fault fires with probability p.
chaos_available()List available fault names.

Observability — entropy.observability

SymbolNotes
Watcher(agent, dataset, trials=20, seed=0).watch(rounds, interval, render), .series(), .anomalies(), .failure_clusters().
render_metric_heatmap(history, path, metrics=None)Writes a PNG heatmap.

Reports & Dashboard — entropy.reports, entropy.dashboard

SymbolNotes
export_report(results, fmt, path, title="EntroPy Report")HTML / JSON / Markdown / PDF / Jupyter.
export(results, fmt, path, title)Lower-level dispatcher.
build_dashboard(results, out_dir), serve_dashboard(out_dir, port=8000)Static dashboard HTML.

Integrations — entropy.integrations

SymbolNotes
from_langchain / from_langgraph / from_openai / from_crewai / from_pydanticai / from_autogen / from_google_adk / from_mcp / from_customWrap a framework agent into a callable.
adapter(name), Adapter, register_adapterDefine custom adapters.
find_adapter(agent), list_adapters()Discovery helpers.

Plugins — entropy.plugins

SymbolNotes
exporter(name=None)Register a report exporter fn(results, path) -> str.
discover(group="entropy.plugins")Import entry-point plugins.