Open-source agent eval coverage

Find what your AI agent evals aren't testing.

Your evals might be passing. Important agent behaviors might still be untested.

Skout Scan analyzes your agent code and existing eval evidence to surface potential testing gaps with the source evidence to investigate them.

Open source · Local-first · No source upload · No API key

Try Skout Scan Watch Demo
$pip install skout-scan
Python 3.12+

See what your evals are missing.

Watch Skout identify an untested Pydantic AI validator failure path, an engineer review the finding and add an eval, and Skout recognize the new coverage.

Watch the complete workflow:

Discover → Review → Add Eval → Verify Resolution

Explore the Pydantic AI example

Actual Skout Scan CLI demonstration.

Quick start

Get started in minutes.

$ pip install skout-scan
$ cd your-agent-repo
$ skout doctor
$ skout scan .
$ skout review
doctor

Check repository compatibility and supported evidence.

scan

Identify potentially untested behaviors.

review

Inspect and classify findings.

If your repository contains generated artifacts you want to ignore, use:skout scan . --exclude "output/**"
Read the GitHub README

Help shape Skout Scan

Trying Skout on a real agent repository? Tell us what worked, what didn't, and whether you found an eval gap worth fixing.

Share Feedback

The problem

Passing evals don't tell you what you're missing.

Agent teams invest in tests, evals, and observability. But passing tests only tell you that the tests you wrote are passing.

They don't necessarily tell you whether those tests cover the important behaviors your agent can perform.

Skout Scan focuses on a different question:

"What important agent behaviors aren't adequately tested?"

Unknown gaps

Important tools, branches, and failure paths may be missing from the eval suite.

Limited visibility

Existing tests don't always make it clear which agent behaviors are actually being verified.

Evolving agents

As agent implementations change, engineers need to revisit whether existing evaluation evidence remains sufficient.

How it works

From agent behavior to eval evidence.

  1. 01

    Discover

    Skout statically extracts supported agent behaviors and existing eval evidence.

    ToolsBranchesFailure pathsWorkflow transitions
  2. 02

    Review

    Skout surfaces high-confidence potential coverage gaps with source references and suggested tests. Engineers review and classify findings.

    add_eval records an intention to add or modify an eval.

  3. 03

    Add Eval

    The engineer adds or modifies an evaluation that exercises the behavior and verifies the expected outcome.

    Skout does not automatically generate the evaluation.

  4. 04

    Verify

    Run Skout again. When Skout detects sufficient new coverage evidence, the original finding can move to:

    coveredresolved

Static analysis. No target code execution. No source upload.

Selecting add_eval does not resolve a finding. Only a rescan with sufficient new coverage evidence does.

Frameworks

Works with popular Python agent frameworks.

LangGraph

Supported static graph transitions, conditional routes, and Python tool patterns.

CrewAI

Supported agents, tasks, tools, crews, and statically analyzable workflow relationships.

Pydantic AI

Supported agent tools, output validators, explicit retry paths, structured outputs, and Pydantic Evals evidence.

Python tools & evals

Supported Python tool patterns, pytest, JSONL, and Python Pydantic Evals Case/Dataset structures.

Supports common statically analyzable patterns. Dynamic construction and unsupported patterns may limit assessment.
View supported patterns and limitations

Skout can analyze supported Python agent/eval code inside a larger mixed-language repository. It does not analyze surrounding Go or other unsupported-language code for behavioral coverage.

Local-first

Built for engineers who care about their code.

Your code stays local

Skout analyzes repositories locally without uploading source code.

No agent execution

Skout does not import or execute code from the target repository.

Explainable findings

Potential gaps include source evidence and suggested tests.

Fits existing workflows

Use Skout alongside your existing tests and evals. It does not replace their execution frameworks.

Examples

Try Skout before scanning your own repo.

Explore small agent examples with intentionally missing eval coverage. Each example includes existing tests, an identifiable potential gap, and a supplied evaluation that demonstrates finding resolution.

LangGraph

Scenario

Refund assistant with an insufficiently tested high-value escalation route.

Explore Example

CrewAI

Scenario

Refund assistant with an uncovered escalation tool invocation.

Explore Example

Pydantic AI

Scenario

Refund assistant with an untested validator retry/failure path.

Explore Example

Building AI agents? Find out what your evals might be missing.

Try Skout Scan on your repository and tell us whether it finds a gap worth testing.

We're especially interested in feedback about false positives, unsupported patterns, and usability.

Try Skout Scan Share Feedback

Runs locally. No source code upload required.