Trust & safety

Evals

Test agents before you ship. Assert facts about a run's trajectory, answer, spend, and governance decisions — and get a policy-coverage report showing which decisions were exercised.

Define cases

evals.py

from antraft import Antraft, ToolRegistry, tool_from_function
from antraft.eval import (
    run_eval, EvalCase,
    assert_tool_called, assert_final_contains, assert_no_denials,
    assert_max_cost, assert_completed,
)
 
def build(task):
    return Antraft.agent(model, ToolRegistry([tool_from_function(add)]), task=task).allow(["add"])
 
report = run_eval(
    EvalCase("math", "what is 17 + 25", [
        assert_tool_called("add"),
        assert_final_contains("42"),
        assert_no_denials(),
        assert_max_cost(0.10),
        assert_completed(),
    ]),
    build,
)
 
print(report.summary())     # pass/fail + policy coverage
assert report.passed

Note

build(task) returns a fresh AntraftBuilder per case; the harness builds, runs, and evaluates it, capturing every governance decision via an internal auditor.

Available assertions

Assertion	Checks
`assert_tool_called(name)`	a tool ran (or `assert_tool_not_called`).
`assert_final_contains(text)`	final answer contains text (or `assert_final_matches` regex).
`assert_no_denials()`	no action was denied.
`assert_max_cost(x)` / `assert_max_tokens(n)`	spend stayed under a cap.
`assert_completed()` / `assert_not_killed()`	the run finished cleanly.
`assert_no_guardrail_block()`	no guardrail blocked content.
`llm_judge(model, criteria)`	an LLM judges the answer against criteria.

Policy coverage

evals.py

report.coverage()      # Counter({"allow": 12, "deny": 2, "pause": 1})
report.passed          # all cases green?
report.pass_count      # number of passing cases

← Previous

Memory

Reliability