Labs experiment
Evals and tracing: every agent run traced, scored and compared before it ships
An evaluation and tracing setup that treats agent quality like a test suite. Every run is traced step by step, answers are scored by an LLM judge against a rubric, saved cases are replayed before each release and the cost of every run is tracked.
Where this pattern fits
- Agents that have to stay reliable while prompts and models change
- Choosing between models or prompts with data instead of guesses
- Release processes that should block regressions
- LangSmith
- Langfuse
- Python
- Pytest
How it works
Select any node to see what it does.
Simulated run
- 00.00cieval suite started on 120 cases
- 02.14judgeaccuracy 0.93, 4 regressions found
- 02.30costmedian run cost down 18 percent
- 02.41cirelease blocked until regressions are fixed
The architecture, step by step
Each part of the diagram, in the order a run moves through it.
- Agent runWhere every run starts.
- TracerCaptures every step of every agent run.
- Span captureRecords each call with its inputs, outputs and timing.
- LLM judgeScores answers against a rubric.
- Regression suiteReplays saved cases before each release.
- Cost monitorTracks tokens and cost per run.
- ToolsThe agents act through the datasets, dashboards and alerts.
How it is built
The design choices that make the pattern hold up with real users.
- Tracing with LangSmith or Langfuse, so every call keeps its inputs, outputs and timing
- A regression suite in Pytest that replays saved cases in CI
- An LLM judge with a written rubric, and a cost monitor per run
- Alerts and dashboards, and a release gate until regressions are fixed
Related
Crewline
A platform that provisions digital employees.
Agentic Workflow
A multi-agent system that runs one business process end to end. From $5,000, 4 to 6 weeks.
Voice receptionist
An AI phone agent that answers, books and hands off to a human.
Knowledge assistant
Answers from company documents, with every claim cited.
CRM tool server
A Model Context Protocol server that gives Claude safe tools for a CRM.