Jev in production › Evaluation and testing

Typed Evals

Python toolkit that sends RAG, response and agent-trace metrics to Jev in one judge request and checks proposed tool calls against policy before they run, with optional threshold calibration against human labels.

Open on GitHub ↗

The source publishes no measured number.
Use
Evaluation and testing
Industry
Developer tools
Form
Open-source tool
Stage
In production
Plugs into
LangChain, CrewAI, Microsoft Agent Framework
Listed
2026-09-24
Found via
github
Repository
TrustifAI/typed_evals
Stars
19
Forks
1
Last push
2026-10-29
Language
Python
License
MIT

The README opens with

Evaluate responses. Guard actions. A Python toolkit for checking LLM answers, evaluating agent runs, and reviewing tool calls before execution

Quickstart · Presets · Tool guards · Calibration · Examples & guides

Give Typed Evals a request, a response, and the evidence used to produce it. Choose what to check, then get a score and pass/fail result for each check. You can also evaluate recorded agent runs and apply policy checks before tools execute.

Badge

For the project's own README, linking back here:

Listed in Jev in production

Also used for evaluation and testing