Metrics Catalog

EntroPy ships 50+ metrics, organized into evaluation layers. The Suite computes a canonical subset by default (marked default); pass metrics=[...] to Suite to select any subset, or call entropy.default_metrics() to list the canonical set.

Bounds

Most metrics return a value in [0, 1]. Higher is not always better — e.g. drift_score, failure_rate and loop_detection are “lower is better”.

Statistical Layer · core

success_rate default

Fraction of trials that succeeded.

reliability default

Mean success across trials.

variance default

Variance of per-trial cost.

robustness default

Mean recovery rate across trials.

confidence_interval default

95% CI for success_rate as [lo, hi] (Wald approx).

entropy default

Normalized entropy over primary actions.

stability_index default

1 − normalized variance of the success outcome.

reliability_score

success_rate × stability_index.

Entropy

action_entropy default

H(actions): randomness of atomic actions.

trajectory_entropy default

H(trajectory): variability of full execution trajectories.

tool_entropy default

H(tool_usage): diversity of tool selection.

robustness_entropy

H(failure distribution): entropy over success/fail/error.

information_gain

Reduction in action uncertainty given success.

exploration_efficiency default

reward / entropy over actions.

Behavioral Layer 4

behavioral_entropy default

Normalized entropy of the primary action per trial.

behavioral_variance default

Variance of the per-trial binary outcome.

trajectory_diversity default

Mean normalized Levenshtein distance across trajectory pairs.

loop_detection

Fraction of trials whose trajectory repeats a state.

goal_stability default

1 − drift; stability of behavior across the trial sequence.

exploration_efficiency_metric

reward / entropy over actions.

emergent_behavior default

Behaviors appearing in ≤5% of trials (rare, potentially emergent).

Reasoning Layer 1

plan_quality

Fraction of trials that produced a reasoning/plan event.

plan_adherence

Fraction of trials whose final action matches a stated plan prefix.

reasoning_consistency

1 − normalized entropy of reasoning step names.

planning_stability

1 − drift of reasoning labels across trials.

goal_decomposition

Average number of distinct sub-steps per trial (normalized).

Tool Layer 2

tool_correctness

Success fraction on tool-using trials.

argument_correctness

Fraction of tool calls carrying non-empty arguments.

tool_reliability default

Success rate over tool-using trials.

tool_selection

Normalized entropy of tool selection.

tool_thrashing

Fraction of trials that called the same tool repeatedly.

hallucinated_tool

Fraction of tool calls flagged as hallucinated.

Execution Layer 3

task_completion

Fraction of trials that completed successfully.

step_efficiency

Mean efficiency 1/(1+events) — fewer steps = more efficient.

latency

Mean latency in seconds (per-trial metadata, else cost proxy).

cost

Mean cost per trial.

retry_rate

Fraction of trials that contained a retry event.

failure_rate

1 − success_rate.

recovery_score default

Fraction of errored trials that still succeeded.

cost_stability

1 − coefficient of variation of per-trial cost.

Memory

memory_correctness

Fraction of memory writes flagged correct.

memory_pollution

Fraction of memory writes flagged polluting/stale.

memory_recall

Success on trials with memory reads.

context_dependency

1 − norm_entropy(actions): rigidity across varied inputs.

Multi-agent

coordination

Success on trials involving a handoff/delegate event.

communication

Mean number of message events per trial.

delegation

Fraction of trials that delegated work.

consensus

1 − normalized entropy of outcomes.

deadlock

Fraction of trials hitting a deadlock/timeout event.

Drift Layer 6

drift_score default

Jensen-Shannon drift of the behavior distribution (first vs second half).

trajectory_drift

Drift over full trajectory signatures.

tool_usage_drift

Drift of the tool-usage distribution.

cost_drift

Normalized absolute drift in mean cost.

behavior_fingerprint

Stable signature of the behavior distribution (top behaviors).

Safety

prompt_injection

Rate of outputs that look like a prompt injection (lower = safer).

instruction_override

Rate of attempts to override system instructions.

pii

Rate of trials leaking PII (email/phone) in output.

toxicity

Rate of trials producing toxic language.

data_leakage

Rate of trials leaking secret-looking material.

Custom metrics

Add your own with the @entropy.metric decorator — see Plugins. Each metric implements compute(self, batch) -> Any and is automatically picked up by the registry.