v0.2.0 · local-first · zero vendor lock-in

AI agent evaluation
and testing for Python

EntroPy (entropy-ai) evaluates AI agent reliability: uncertainty, drift, stochastic behavior, and emergent failures — via Monte Carlo evaluation over repeated runs and entropy-based metrics. Framework-agnostic, runs locally, no vendor lock-in.

Get Started → Install from PyPI GitHub ↗
🎲

Monte Carlo by default

Run any agent N times and aggregate behavior across trials — not a single lucky pass.

📊

50+ entropy metrics

Success, reliability, behavioral & trajectory entropy, drift, robustness, safety and more.

🌊

Drift detection

Track how an agent's behavior distribution shifts across trials and over time.

🔥

Chaos engineering

Inject faults (API failures, timeouts, tool crashes) and measure resilience.

🧩

Framework agnostic

Adapters for LangChain, LangGraph, OpenAI, CrewAI, PydanticAI, AutoGen, Google ADK, MCP — all lazy.

🖥️

Local-first

Reports, dashboards, observability and a CLI — everything runs on your machine.

Install

pip install entropy-ai                # core
pip install "entropy-ai[langchain]"   # optional: a framework adapter
pip install "entropy-ai[pdf,jupyter]" # optional: PDF / Jupyter reports

Your first evaluation

from entropy import evaluate, assert_stable

def my_agent(text):
    return text.strip().lower()

results = evaluate(my_agent, {" Hello ": "hello", "WORLD": "world"}, trials=20)
print(results["success_rate"])        # 1.0
print(results["behavioral_entropy"])  # 0.0
assert_stable(results, success_rate__gte=0.95)

Test agent reliability before your next release

A single successful answer does not tell you how often an agent will succeed. EntroPy runs your synchronous Python callable repeatedly and measures the distribution of its outputs and recorded events. Use expected answers or a custom check to define success.

Start locally, then connect your agent

The example above needs no API key. Install with pip install entropy-ai, save the code in a Python file, and run it. Two inputs with 20 trials make 40 agent calls. When you connect a paid provider, those calls may incur costs.

Choose what to measure

Measure output consistency with behavioral entropy, compare repeated runs for drift, and use assert_stable to fail a regression check when success falls below your threshold. Without expected answers or a check, success only means no reported error. Tool and trajectory metrics need recorded events; a seed does not make a remote model deterministic.