EntroPy (entropy-ai) evaluates AI agent reliability: uncertainty, drift,
stochastic behavior, and emergent failures — via Monte Carlo
evaluation over repeated runs and entropy-based metrics.
Framework-agnostic, runs locally, no vendor lock-in.
Run any agent N times and aggregate behavior across trials — not a single lucky pass.
Success, reliability, behavioral & trajectory entropy, drift, robustness, safety and more.
Track how an agent's behavior distribution shifts across trials and over time.
Inject faults (API failures, timeouts, tool crashes) and measure resilience.
Adapters for LangChain, LangGraph, OpenAI, CrewAI, PydanticAI, AutoGen, Google ADK, MCP — all lazy.
Reports, dashboards, observability and a CLI — everything runs on your machine.
pip install entropy-ai # core
pip install "entropy-ai[langchain]" # optional: a framework adapter
pip install "entropy-ai[pdf,jupyter]" # optional: PDF / Jupyter reports
from entropy import evaluate, assert_stable
def my_agent(text):
return text.strip().lower()
results = evaluate(my_agent, {" Hello ": "hello", "WORLD": "world"}, trials=20)
print(results["success_rate"]) # 1.0
print(results["behavioral_entropy"]) # 0.0
assert_stable(results, success_rate__gte=0.95)
A single successful answer does not tell you how often an agent will succeed. EntroPy runs your synchronous Python callable repeatedly and measures the distribution of its outputs and recorded events. Use expected answers or a custom check to define success.
The example above needs no API key. Install with pip install entropy-ai,
save the code in a Python file, and run it. Two inputs with 20 trials make 40 agent calls.
When you connect a paid provider, those calls may incur costs.
Measure output consistency with behavioral entropy, compare repeated runs for drift,
and use assert_stable to fail a regression check when success falls below
your threshold. Without expected answers or a check, success only means no reported error.
Tool and trajectory metrics need recorded events; a seed does not make a remote model deterministic.