Core Concepts
EntroPy treats agent behavior as a distribution, not a single output. The central idea: run an agent many times, record what happened each time, then summarize the distribution with entropy-based metrics.
AgentRun
One execution of an agent on a single input. An agent may return an
AgentRun directly, or EntroPy wraps a plain return value
into one.
| Field | Type | Meaning |
|---|---|---|
run_id | str | Unique id for the run |
input | Any | The input the agent received |
output | Any | The agent's output |
events | list[Event] | Recorded behavior (actions, tools, reasoning…) |
cost | float | Cost of the run (optional) |
metadata | dict | Latency, errors, custom data |
Events
Behavior is captured as a sequence of typed Events. The
built-in event types carry semantic meaning for metrics:
| Event | type | Used by |
|---|---|---|
ActionEvent | action | Primary action label, entropy, drift |
ReasoningEvent | reasoning | Plan quality, consistency |
ToolCallEvent | tool | Tool reliability, selection, thrashing |
ObservationEvent | observation | Behavior trajectories |
MemoryReadEvent / MemoryWriteEvent | memory_read / memory_write | Memory metrics |
ErrorEvent | error | Failure / recovery / robustness |
StateTransitionEvent | state | Trajectory graph |
Trace & Graph
A Trace collects runs plus an optional behavior graph
(Node / Edge). It exposes exporters:
json(), df(), graph(),
otel(), and zip(path).
Suite & Batch
The Suite runs an agent trials times per case
and produces a Batch per input. Each Batch
summarizes one input's trials: outputs, behaviors
(trajectory signatures), actions, tool_calls,
successes, costs, errored,
recovered. Metrics consume batches; the suite averages each
metric across inputs to produce the final result dictionary.
Results dictionary
Suite.run() returns a flat dict of metric
name → value (or list for confidence_interval and the
emergent_behavior / behavior_fingerprint
signatures). Numeric values are in [0, 1] where a bound
applies.
Evaluation layers
Metrics are organized into six evaluation layers (see Metrics Catalog):
- Reasoning — plan quality, adherence, consistency
- Tool — correctness, reliability, selection
- Execution — completion, latency, cost, recovery
- Behavioral — entropy, variance, drift, loops
- Robustness — resilience to failure
- Drift — long-term behavior change