Evals are offline, maintainer-facing test fixtures for checking coding-agent behavior. They let you validate that the agent produces correct results for specific task types — without calling a live model (or with one, depending on the mode).
An eval suite is a JSON file defining a set of tasks. Each task has:
The harness materializes the fixture, runs the agent, and scores the result.
{
"id": "my-eval-suite",
"name": "My Evaluation Suite",
"description": "Validates core coding-agent behaviors",
"tasks": [ ... ]
}
{
"id": "add-health-endpoint",
"name": "Add Health Endpoint",
"description": "Add a GET /health endpoint with DB and cache checks",
"tags": ["api", "backend"],
"difficulty": "medium",
"prompt": "Add a GET /health endpoint to the API server that checks database and cache connectivity. Return {status, db, cache} JSON.",
"workspaceFixture": "fixtures/api-server",
"verificationCommands": [
{
"id": "build",
"name": "Build the project",
"command": ["go", "build", "./..."]
},
{
"id": "test",
"name": "Run tests",
"command": ["go", "test", "./internal/server/..."]
}
],
"expectedChangedFiles": [
"internal/server/routes.go",
"internal/server/handlers.go"
],
"forbiddenChangedFiles": [
"go.mod",
"go.sum",
"package.json"
],
"contextChecks": {
"requiredFiles": [
"internal/server/handlers.go",
"internal/server/routes.go"
],
"forbiddenFiles": [
"node_modules/cache.txt"
]
},
"requiredTraceEvents": [
"tool:read_file",
"tool:edit_file",
"verify:build",
"verify:test"
]
}
| Field | Required | Description |
|---|---|---|
id |
Yes | Unique task identifier (kebab-case) |
name |
Yes | Human-readable task name |
description |
No | Longer description |
tags |
No | Labels for filtering (e.g. ["api", "backend"]) |
difficulty |
No | easy, medium, hard |
prompt |
Yes | The instruction given to the agent |
workspaceFixture |
Yes | Path to the fixture directory (relative to suite file) |
verificationCommands |
Yes | Array of commands to verify the result |
expectedChangedFiles |
No | Files that SHOULD be modified |
forbiddenChangedFiles |
No | Files that should NOT be touched |
contextChecks |
No | Required/forbidden files in the workspace |
requiredTraceEvents |
No | Expected tool-call events in the agent’s output trace |
{
"id": "test",
"name": "Run tests",
"command": ["go", "test", "./internal/server/..."]
}
| Field | Required | Description |
|---|---|---|
id |
Yes | Unique command identifier |
name |
Yes | Human-readable name |
command |
Yes | Command + arguments as an array |
Each command has a 10-minute timeout (defaultCommandTimeout).
pvyai eval --suite evals/my-suite.json
Parses the suite JSON, checks for schema errors, validates task definitions, and reports the task count. Does NOT copy fixtures, invoke an agent, or run verification commands.
Use this to check your suite is well-formed before running it.
pvyai eval run \
--suite evals/my-suite.json \
--task add-health-endpoint \
--workspace /tmp/prepared-workspace
Scores an already-mutated git workspace. Runs the verification commands, collects changed files via git status --porcelain, and emits the report. Does NOT copy fixtures or invoke an agent.
Use this when you’ve already run the agent manually and want to score its output.
pvyai eval bench \
--suite evals/my-suite.json \
--task add-health-endpoint \
--work-root /tmp/pvyai-evals \
--agent-command "pvyai exec --cwd {workspace} {prompt}" \
--timeout 600s \
--keep-workspaces
The full pipeline:
The --agent-command supports these placeholders:
| Placeholder | Replaced with |
|---|---|
{prompt} |
The task’s prompt text |
{workspace} |
The copied fixture workspace path |
{task_id} |
The selected task ID |
{model} |
Current model ID (for model-matrix runs) |
Run the same suite across multiple models to compare:
pvyai eval bench \
--suite evals/my-suite.json \
--agent-command "pvyai exec --cwd {workspace} --model {model} {prompt}" \
--model claude-sonnet-4.5 \
--model claude-opus-4.1 \
--model gemini-2.5-pro \
--report-dir /tmp/eval-reports
Or comma-separated:
--models claude-sonnet-4.5,claude-opus-4.1,gemini-2.5-pro
| Status | Meaning |
|---|---|
pass |
All checks passed |
fail |
At least one check failed |
blocked |
The harness couldn’t run the task or collect inputs |
error |
The suite, task ID, or command couldn’t be interpreted |
| Kind | What it checks |
|---|---|
command |
Did the verification command exit 0? |
changed_files |
Do the changed files match expectedChangedFiles? |
context |
Are required files present? Are forbidden files absent? |
trace |
Are required trace events present in the agent’s output? |
The harness parses JSONL trace events from the agent’s stdout. Events are keyed as type:name:
| Event key | Meaning |
|---|---|
tool:read_file |
The agent called read_file |
tool:edit_file |
The agent called edit_file |
tool:exec_command |
The agent called exec_command |
verify:go-test |
A verification command named “go-test” ran |
verify:build |
A verification command named “build” ran |
Example requiredTraceEvents:
"requiredTraceEvents": [
"tool:read_file",
"tool:edit_file",
"verify:build",
"verify:test"
]
This verifies the agent actually read files, edited files, and that the build and test commands ran — not just that the final state is correct.
The report follows the pvyai.agenteval.report.v1 contract:
{
"contract": "pvyai.agenteval.report.v1",
"suite": { "id": "my-eval-suite", "name": "..." },
"summary": {
"total": 3,
"passed": 2,
"failed": 1,
"blocked": 0,
"errors": 0
},
"results": [
{
"id": "add-health-endpoint",
"name": "Add Health Endpoint",
"status": "pass",
"checks": [
{
"id": "build",
"name": "Build the project",
"kind": "command",
"status": "pass",
"exitCode": 0
},
{
"id": "changed-files",
"name": "Changed files",
"kind": "changed_files",
"status": "pass",
"expectedFiles": ["internal/server/routes.go"],
"actualFiles": ["internal/server/routes.go"],
"missingFiles": [],
"unexpectedFiles": []
}
]
}
]
}
When running locally, the agent runs with its own sandbox guardrails:
The sandbox engine classifies every shell command:
| Risk level | Patterns | Action |
|---|---|---|
| Destructive | rm -rf /, mkfs, dd if=, recursive chmod 777 of root, fork bombs, raw block device writes |
Always denied (even in unsafe mode) |
| High risk | git push --force, DROP TABLE, docker rm, etc. |
Prompt in all modes |
| Normal | Standard build/test/edit commands | Follow permission mode |
Behavioral guardrails prevent runaway agents:
| Rule | Threshold | Effect |
|---|---|---|
| Empty turns | 3 consecutive | Stop the run |
| Tool failure loop | 6 consecutive same-error failures | Stop the run |
| Stale plan | 10 tool calls or 8 turns without update_plan update |
Inject reminder |
| Completion nudges | 3 max | Bound headless re-prompts |
When the coding agent is routed through the PVY.ai Platform Open WebUI proxy, an additional security layer is active: the PVYai Sentinel.
Note: The Sentinel lives in the Open WebUI middleware (
backend/open_webui/utils/middleware.py), not in the coding agent’s Go binary. It intercepts the SSE stream between the model and the TUI. Not all models have the Sentinel active — an admin can selectively enable/disable it per model preset in Open WebUI.
Model (LLM)
│
▼ SSE stream (tool_calls, text, etc.)
┌──────────────────────────────────────┐
│ PVYai Sentinel (middleware.py) │
│ │
│ 1. inlet(): pre-request inspection │
│ - Wing selection flags │
│ - Handover offers │
│ - Destructive command detection │
│ - AgentWorld simulation │
│ │
│ 2. SSE interception: mid-stream │
│ - Regex patterns match │
│ destructive bash tool calls │
│ - Replace with safety warning │
│ - LLM never executes the command │
│ │
│ 3. outlet(): post-response analysis │
│ - Token usage tracking │
│ - Context threshold detection │
│ - Finding detection for memory │
│ - Session handover generation │
└──────────────────────────────────────┘
│
▼ Filtered SSE stream
OpenCode Desktop / TUI
The Sentinel intercepts these destructive patterns in bash tool calls:
| Category | Patterns |
|---|---|
| Git | git branch -D, git push --force/-f, git reset --hard |
| Filesystem | rm -rf, shred, > /dev/sd[a-z], mkfs |
| Docker | docker rm/rmi/stop/kill |
| System | systemctl stop/disable/mask, kill -9 |
| Database | DROP TABLE/DATABASE, TRUNCATE, DELETE FROM |
| Package | pip uninstall, chmod -R 0, chown -R root |
Commit messages are stripped before matching to prevent false positives (e.g. git commit -m "message containing reset --hard" doesn’t trigger the git reset --hard pattern).
| Aspect | Local sandbox (Go binary) | PVYai Sentinel (middleware) |
|---|---|---|
| Where | In the coding agent process | In the Open WebUI proxy |
| When | Before tool execution | Before the TUI receives the tool call |
| What | Regex + risk classification → prompt/deny | Regex → replace with safety warning |
| Scope | All bash and exec_command tool calls |
All bash tool calls in the SSE stream |
| Config | Permission mode (ask/auto/unsafe) | Per-model enable/disable (admin) |
| Active | Always (in the Go binary) | Only when routed through Open WebUI + admin enabled |
When running evals through the PVY.ai Platform proxy:
requiredTraceEvents should account for this — if a destructive command is expected (e.g. testing rm -rf handling), the eval should either:
In short, Software Engineers / Developers are halted to monitor in the beginning the behaviour of Interference in Loops with Swarms and Specialist (your own defined Sub-agents) and if PVYai Sentinal is causing to much breaks in a anyway typically isolated environment, collect this findings, analyse and report it back. Or we define the same Models without PVYai Sentinel oversight to be picked over the Model Selector then.
With LLMs boasting extensive context windows (around 1 million tokens), a limit of 50 tool calls might prove insufficient. PVYai Sentinel offers a solution by actively monitoring LLM context saturation in real-time. It proactively manages the context, removing less relevant information to make room for new data, thereby preventing performance degradation. This dynamic context management system functions as intended within the OpenCode Desktop environment. If this is beneficially for PVYai Agent in Loops in a Swarm and Team of Agents setup, has to be evualated, and maybe also differently implemented or entirely disabeld.
{
"id": "add-format-function",
"name": "Add Format Function",
"difficulty": "easy",
"prompt": "Add a function `FormatDuration(d time.Duration) string` to internal/utils/format.go that formats a duration as a human-readable string (e.g. '2h3m45s'). Add unit tests.",
"workspaceFixture": "fixtures/empty-project",
"verificationCommands": [
{
"id": "test",
"name": "Run tests",
"command": ["go", "test", "./internal/utils/..."]
}
],
"expectedChangedFiles": [
"internal/utils/format.go",
"internal/utils/format_test.go"
],
"forbiddenChangedFiles": [
"go.mod",
"go.sum"
]
}
{
"id": "fix-nil-pointer",
"name": "Fix Nil Pointer Bug",
"difficulty": "medium",
"prompt": "There's a nil pointer dereference in internal/server/handle.go when the request body is empty. Find and fix it. Add a test that sends an empty body and verifies no panic.",
"workspaceFixture": "fixtures/buggy-server",
"verificationCommands": [
{
"id": "build",
"name": "Build",
"command": ["go", "build", "./..."]
},
{
"id": "test",
"name": "Run tests",
"command": ["go", "test", "-run", "TestHandleEmpty", "./internal/server/..."]
}
],
"expectedChangedFiles": [
"internal/server/handle.go",
"internal/server/handle_test.go"
],
"requiredTraceEvents": [
"tool:read_file",
"tool:grep",
"tool:edit_file",
"verify:test"
]
}
{
"id": "refactor-to-interfaces",
"name": "Refactor to Interfaces",
"difficulty": "hard",
"prompt": "Refactor the database access layer to use interfaces instead of concrete types, enabling mocking for tests. Keep all existing tests passing. The interface should be in internal/db/interface.go.",
"workspaceFixture": "fixtures/db-project",
"verificationCommands": [
{
"id": "build",
"name": "Build",
"command": ["go", "build", "./..."]
},
{
"id": "test",
"name": "All tests pass",
"command": ["go", "test", "./..."]
},
{
"id": "lint",
"name": "Lint",
"command": ["golangci-lint", "run", "./internal/db/..."]
}
],
"expectedChangedFiles": [
"internal/db/interface.go",
"internal/db/conn.go"
],
"forbiddenChangedFiles": [
"go.mod"
],
"contextChecks": {
"requiredFiles": ["internal/db/interface.go"],
"forbiddenFiles": ["internal/db/debug.go"]
}
}
# Validate the suite schema
pvyai eval --suite evals/my-suite.json
# Validate all suites in a directory
pvyai eval --suite evals/
# Score a single task
pvyai eval run \
--suite evals/my-suite.json \
--task add-format-function \
--workspace /tmp/my-workspace
# Output as JSON
pvyai eval run \
--suite evals/my-suite.json \
--task add-format-function \
--workspace /tmp/my-workspace \
--json
# Run a single task
pvyai eval bench \
--suite evals/my-suite.json \
--task add-format-function \
--work-root /tmp/evals \
--agent-command "pvyai exec --cwd {workspace} {prompt}"
# Run all tasks
pvyai eval bench \
--suite evals/my-suite.json \
--work-root /tmp/evals \
--agent-command "pvyai exec --cwd {workspace} {prompt}"
# Model matrix
pvyai eval bench \
--suite evals/my-suite.json \
--work-root /tmp/evals \
--agent-command "pvyai exec --cwd {workspace} --model {model} {prompt}" \
--model claude-sonnet-4.5 \
--model claude-opus-4.1 \
--report-dir /tmp/eval-reports
# Keep workspaces for inspection
pvyai eval bench \
--suite evals/my-suite.json \
--work-root /tmp/evals \
--agent-command "pvyai exec --cwd {workspace} {prompt}" \
--keep-workspaces
# Set a per-task timeout
pvyai eval bench \
--suite evals/my-suite.json \
--work-root /tmp/evals \
--agent-command "pvyai exec --cwd {workspace} {prompt}" \
--timeout 300s
PVYai’s eval harness is a static, offline scoring tool — it compares a captured run against a fixed rubric. There is no built-in feedback loop that updates prompts, model selection, or suite definitions based on past results.
However, you can implement self-learning patterns externally:
#!/bin/bash
# eval-feedback-loop.sh
REPORT=$(pvyai eval bench --suite evals/regression.json \
--agent-command "pvyai exec --cwd {workspace} {prompt}" \
--report-dir /tmp/reports \
--json)
FAILED=$(echo "$REPORT" | jq '.summary.failed')
if [ "$FAILED" -gt 0 ]; then
echo "=== Failed tasks ==="
echo "$REPORT" | jq '.results[] | select(.status == "fail") | .name'
# Add failing tasks to the next eval cycle's priority
echo "$REPORT" | jq -r '.results[] | select(.status == "fail") | .id' \
>> evals/regression-priority.txt
# Slack notification
curl -X POST "$SLACK_WEBHOOK" -d "text=Eval failures: $FAILED"
fi
Run evals, identify failure patterns, and refine your skills’ prompts:
# Run evals with the current skill set
pvyai eval bench --suite evals/quality.json \
--agent-command "pvyai exec --cwd {workspace} {prompt}" \
--report-dir /tmp/reports
# Analyze failures
jq '.results[] | select(.status == "fail") | {
task: .name,
failures: [.checks[] | select(.status != "pass") | .message]
}' /tmp/reports/report.json
# Update the relevant skill based on failure patterns
# (manual or scripted)
$EDITOR ~/.local/share/pvyai/skills/coding-standards/SKILL.md
# Re-run to verify improvement
pvyai eval bench --suite evals/quality.json \
--agent-command "pvyai exec --cwd {workspace} {prompt}" \
--report-dir /tmp/reports-v2
# Compare
diff <(jq -S '.summary' /tmp/reports/report.json) \
<(jq -S '.summary' /tmp/reports-v2/report.json)
Use the model matrix to find the best model per task type:
pvyai eval bench \
--suite evals/regression.json \
--agent-command "pvyai exec --cwd {workspace} --model {model} {prompt}" \
--model claude-sonnet-4.5 \
--model claude-haiku-4.5 \
--model gemini-2.5-flash \
--report-dir /tmp/model-comparison
# Find the model with the highest pass rate
jq -s '
group_by(.model) |
map({
model: .[0].model,
pass_rate: (map(select(.status == "pass")) | length) / length
}) |
sort_by(-.pass_rate)
' /tmp/model-comparison/*/results.json
When routing through the PVYai Platform with the Sentinel active:
forbiddenChangedFiles and requiredTraceEvents to account for Sentinel behaviorevals/
├── regression.json # Suite file
├── quality.json # Another suite
├── fixtures/
│ ├── empty-project/ # Minimal Go project
│ │ ├── go.mod
│ │ └── main.go
│ ├── buggy-server/ # Server with a nil pointer bug
│ │ ├── go.mod
│ │ ├── internal/
│ │ │ └── server/
│ │ │ ├── handle.go
│ │ │ └── handle_test.go
│ │ └── ...
│ └── db-project/ # Database access project
│ ├── go.mod
│ ├── internal/db/
│ └── ...
└── reports/ # Generated reports
├── report-20260707.json
└── report-20260708.json
| Practice | Why |
|---|---|
Start with validate mode |
Catch schema errors before running expensive benches |
Use forbiddenChangedFiles |
Prevent scope creep — ensure the agent doesn’t touch go.mod etc. |
Use requiredTraceEvents |
Verify the agent actually used the right tools, not just got lucky |
Run bench with --keep-workspaces |
Inspect the agent’s work when it fails |
| Use the model matrix | Different models excel at different task types |
| Keep fixtures minimal | Smaller fixtures = faster evals = more iterations |
| Tag tasks by difficulty | Track which difficulty levels your models handle well |
| Don’t use evals as release gates | They’re quality signals, not pass/fail gates (per the official guidance) |
| Compare across runs | Use --report-dir and diff reports to track improvement |