zzstoatzz.io
zig langref cli
nate.tngl.io/zigman
evals: go all-in on Pi, drop pydantic-ai + opencode
the loop is now one stdlib-only file (evals/run.py) orchestrating two CLIs:
Pi (@earendil-works/pi-coding-agent, headless `-p --mode json`, native --skill)
as the agent, and `zig test` as the objective oracle. gemma-4 via mlx_vlm.server
returns real structured tool_calls (the thing ollama+gemma3/qwen never did), so a
competent local agent finally drives the loop end-to-end.
removes eval.py (pydantic-ai), opencode_runner.py, prove.py — opencode's overhead
was what broke weak models; Pi (lighter) covers every role we used it for.
DESIGN.md updated to the Pi/gemma-4/MLX stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
author
zzstoatzz
co-author
Claude Opus 4.8 (1M context)
date
(Jun 15, 2026, 2:10 PM -0500)
commit
86f06c11
86f06c11d453b0b418552a8d284d5a2b925f8c34
parent
b8ddf403
b8ddf4033c16e3b223d45cd6a63da5ac7199e486
+315
-804
+43
-73
DESIGN.md
+43
-73
DESIGN.md
···
34
34
as long as they're not *too* dumb to use a tool at all; the interesting question
35
35
then becomes "how do you enable a not-very-smart model to use a tool effectively."
36
36
37
-
### model-agnostic by construction
38
-
39
-
the harness is provider-agnostic (prefect-mcp-server's pattern). a small registry
40
-
maps `provider -> (base_url, api_key_env)`; every provider is OpenAI-compatible,
41
-
so one code path drives a dumb local model and a smart cloud one alike. model
42
-
*roles* are independent specs:
43
-
44
-
- `--agent-model ollama/gemma3-tools` — the model under test (dumb→smart)
45
-
- `--judge-model openai/gpt-4o` — the evaluator (pair a dumb agent with a smart judge)
46
-
47
-
a bare name defaults to local ollama. this lets us sweep the same corpus across a
48
-
ladder of models to find where tool-use enablement kicks in.
49
-
50
37
## why zig is a good domain for this
51
38
52
39
unlike most agent evals, zig has an **objective oracle: the compiler.** we do not
53
-
need an LLM to grade whether code is correct — `zig test` is ground truth (it
54
-
compiles and the hidden test passes, or it doesn't). that removes the usual
40
+
need an LLM to grade whether code is correct — `zig test` is ground truth: it
41
+
compiles *and runs* the agent's function against a hidden test (oracle inputs are
42
+
forced to runtime, so it executes, not just type-checks). that removes the
55
43
"who judges the judge" problem from the core signal.
56
44
57
-
we *also* use an LLM judge, but for a different question (below).
58
-
59
45
## architecture of the eval
60
46
61
-
modeled on [prefect-mcp-server](https://github.com/PrefectHQ/prefect-mcp-server)'s
62
-
evals (agent-with-tools + an llm judge) and on the author's
63
-
[does-it-tool](https://tangled.org/zzstoatzz.io/does-it-tool) (pydantic-ai over any
64
-
OpenAI-compatible endpoint; the serving layer, not the model, is usually what
65
-
breaks small-model tool use).
66
-
67
-
`evals/eval.py` runs one or more **tasks**. each task is a real one-shot coding
68
-
ask (e.g. "implement `pub fn cstrLen(s: [*:0]const u8) usize`"), plus a hidden
69
-
oracle `test {}` and a known-good reference solution.
47
+
the stack is all local, no GUI, and each layer is a CLI we orchestrate from
48
+
stdlib python (`evals/run.py`) — no agent framework of our own:
70
49
71
-
**eval agent** — a pydantic-ai agent with three tools:
72
-
- `write_file(filename, content)` — code flows through a tool *argument*, not
73
-
fenced prose (this matters; see findings).
74
-
- `zig_test(filename)` — compiles and runs the file, returns compiler output so
75
-
the agent can read errors and iterate.
76
-
- `zigman(query)` — reads a langref section. the agent is **not told to use it**;
77
-
whether it reaches for it is part of what we observe.
78
-
79
-
**objective layer** — after the run, we compile the agent's final file against
80
-
the hidden oracle test. pass/fail, no opinion.
81
-
82
-
**judge agent** — a second LLM sees the task, the full tool-call trace, the
83
-
agent's final message, and the objective result, and returns a structured verdict
84
-
(`passed: bool`, `reasoning: str`): did the agent do a *good job*? this captures
85
-
quality the binary oracle can't — did it verify with `zig_test`, did it consult
86
-
the reference when genuinely unsure, did it solve the actual task vs. game a test.
50
+
- **agent harness: [Pi](https://github.com/badlogic/pi-mono)** (`@earendil-works/pi-coding-agent`),
51
+
run headless (`pi -p --mode json`). lightweight built-in `read`/`bash`/`edit`/`write`
52
+
tools and native agentskills.io `--skill` loading. we parse its JSON event stream
53
+
for the tool-call trace.
54
+
- **model: gemma-4-12B** served OpenAI-compatible by `mlx_vlm.server` on `:1234`
55
+
(LM Studio drops in on the same port). model-agnostic: any Pi provider/model works,
56
+
so we can sweep dumb→smart. gemma-4 is the generation that finally tool-calls *and*
57
+
writes competent zig locally (gemma3/qwen2.5 via ollama did not — see findings).
58
+
- **objective oracle:** after the run we read the agent's `solution.zig`, append the
59
+
hidden oracle `test`, and `zig test`. pass/fail, no opinion. `selftest` validates
60
+
every oracle against a reference solution before any model runs.
87
61
88
-
`selftest` validates every oracle against its reference solution before any model
89
-
runs, so a broken oracle can't silently invalidate the eval.
62
+
each task is a real one-shot coding ask (e.g. "implement `pub fn cstrLen(...)`") plus
63
+
a hidden oracle test and a reference solution. **ablation:** the zigman arm loads
64
+
`skills/zigman` via `--skill` and has the `zigman` CLI on PATH (the agent runs it via
65
+
`bash`); the bare arm doesn't. lift = pass-rate delta.
90
66
91
67
## current state (what's proven)
92
68
93
-
- the pipeline runs end-to-end against local gemma3-tools: agent calls
94
-
`write_file` + `zig_test`, code compiles, the hidden oracle passes, and the judge
95
-
returns a structured PASS. a committed sample trace lives at
96
-
[`evals/sample_report.json`](evals/sample_report.json); live runs write to
97
-
`evals/reports/` (gitignored — machine-specific).
69
+
- the pipeline runs end-to-end, fully local: Pi + gemma-4 (MLX) writes `solution.zig`,
70
+
runs `zig test` itself via `bash`, and the hidden oracle passes. a committed sample
71
+
trace lives at [`evals/sample_report.json`](evals/sample_report.json); live runs
72
+
write to `evals/reports/` (gitignored).
98
73
- the corpus is seven intentionally reference-dependent tasks (`@intCast` single-arg
99
74
form, `for (xs, 0..) |x, i|` index capture, exhaustive enum switch, labeled-block
100
-
break, error-union `try`, sentinel array slicing, sentinel pointer len). every
101
-
oracle is validated against a reference solution by `selftest` before any model
102
-
runs.
103
-
- the agentskills.io skill loads via `pydantic_ai_skills.SkillsToolset`, the same
104
-
mechanism the author's phi bot uses.
75
+
break, error-union `try`, sentinel array slicing, sentinel pointer len), all
76
+
selftest-validated.
105
77
106
-
### findings that unblocked it (serving layer, per does-it-tool's thesis)
78
+
### the two-day detour, and what it taught us (serving layer, per does-it-tool's thesis)
107
79
108
-
1. **code must go through a tool argument.** with no file/exec tool, the model
109
-
emits code as a fenced ```` ```json ```` / ```` ```zig ```` block in its reply;
110
-
ollama's gemma3-tools parser drops fenced tool calls, so nothing executes. give
111
-
it `write_file` and the call parses cleanly.
112
-
2. **terse, imperative prompting.** a verbose, tutorial-style system prompt makes
113
-
weak models *narrate* tool calls ("first I would call write_file…") instead of
114
-
emitting them. "use the tools, don't describe them; start by calling
115
-
write_file" fixes it.
116
-
3. gemma3-tools handles simple tool calls reliably (does-it-tool passes 6/6) but
117
-
is fragile on code-bearing calls; the fixes above are what make it work.
118
-
4. **adding the skill re-broke it.** loading `skills/zigman/SKILL.md` via
119
-
`SkillsToolset` (the realistic product inducement) added enough prompt/tool
120
-
overhead that gemma reverted to fencing its `write_file` call — it scored 0/7
121
-
and *still never called zigman*. so on this model the honest "skill + CLI"
122
-
product currently *hurts*: the weak model can emit clean tool calls **or**
123
-
absorb skill instructions, not both. this is the central open problem — likely
124
-
needs a stronger (but still cheap) model, a lighter inducement, or a serving
125
-
layer tuned to stay robust under longer prompts.
80
+
getting a *local* model to use tools at all was the whole fight, and the lesson is
81
+
that it's a **serving-layer / model-generation** problem, never "local can't":
82
+
83
+
1. **ollama + gemma3/qwen2.5 was a dead end for code-bearing tool calls.** gemma3-tools
84
+
is a prompt *hack* (no native tool-calling; a Modelfile coerces bare JSON that
85
+
ollama string-parses) and it fences code-bearing calls; stock qwen2.5-coder emits
86
+
bare JSON ollama won't parse; llama3.1 narrates. each a different per-model quirk.
87
+
2. **opencode and `SkillsToolset` made it worse** — their large prompt/tool overhead
88
+
destabilized weak models' tool formatting. lean harness wins.
89
+
3. **the fix was the engine+model, not the harness** (this is Vicki Boykis's "local
90
+
models are good now" point): **gemma-4 served by mlx_vlm returns proper structured
91
+
`tool_calls`** — immune to fencing because the call isn't text — and writes
92
+
competent zig. swap the serving layer, don't hand-tune Modelfiles.
93
+
94
+
so "how do you enable a not-too-dumb model to use a tool" is overwhelmingly a plumbing
95
+
question (engine, template, tool-call format), not an IQ one.
126
96
127
97
## the two loops
128
98
-413
evals/eval.py
-413
evals/eval.py
···
1
-
# /// script
2
-
# requires-python = ">=3.10"
3
-
# dependencies = ["pydantic-ai>=1.0"]
4
-
# ///
5
-
"""agentic zig eval: bare vs. zigman ablation, compiler oracle, optional llm judge.
6
-
7
-
the eval agent is a real coding agent with tools — `write_file` (code goes
8
-
through a tool arg, never fenced prose), `zig_test` (compile + run, read errors,
9
-
iterate), and (in the zigman arm) `zigman` (read the language reference). it is
10
-
NOT told to use the reference; whether it reaches for it is observed.
11
-
12
-
the **objective oracle** is `zig test` against a hidden test — ground truth, no
13
-
opinion. we run each task two ways (bare / with-zigman) and report the pass-rate
14
-
delta. tasks are intentionally reference-dependent: the goal is a corpus where a
15
-
weak model fails bare but succeeds with zigman, so lift is measurable.
16
-
17
-
an optional **llm judge** (--judge) scores the with-zigman trace for quality the
18
-
binary oracle can't see (did it verify, did it consult the reference when stuck).
19
-
20
-
usage:
21
-
uv run evals/eval.py selftest # validate oracles, no model
22
-
uv run evals/eval.py run --model gemma3-tools # bare vs zigman, all tasks
23
-
uv run evals/eval.py run --model gemma3-tools -k intcast --judge
24
-
uv run evals/eval.py run --model gemma3-tools --arms zigman --trials 3
25
-
"""
26
-
27
-
from __future__ import annotations
28
-
29
-
import argparse
30
-
import asyncio
31
-
import json
32
-
import os
33
-
import subprocess
34
-
import sys
35
-
import tempfile
36
-
from dataclasses import dataclass, field
37
-
from pathlib import Path
38
-
39
-
from pydantic import BaseModel
40
-
from pydantic_ai import Agent, RunContext
41
-
from pydantic_ai.models.openai import OpenAIChatModel
42
-
from pydantic_ai.providers.openai import OpenAIProvider
43
-
from pydantic_ai.usage import UsageLimits
44
-
45
-
46
-
# --- model-agnostic provider registry (prefect-mcp-server style) ----------
47
-
# every provider here is OpenAI-compatible, so one code path drives dumb local
48
-
# models and smart cloud ones alike. a model spec is "provider/model"; a bare
49
-
# name defaults to local ollama. roles (agent / judge) are independent specs, so
50
-
# you can pair a dumb agent with a smart judge.
51
-
52
-
PROVIDERS: dict[str, tuple[str, str | None]] = {
53
-
# provider: (base_url, api_key_env or None)
54
-
"ollama": (os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434/v1"), None),
55
-
"openai": ("https://api.openai.com/v1", "OPENAI_API_KEY"),
56
-
"openrouter": ("https://openrouter.ai/api/v1", "OPENROUTER_API_KEY"),
57
-
"groq": ("https://api.groq.com/openai/v1", "GROQ_API_KEY"),
58
-
}
59
-
60
-
61
-
def make_model(spec: str) -> OpenAIChatModel:
62
-
"""resolve a "provider/model" spec to a model; bare name => local ollama."""
63
-
provider, _, name = spec.partition("/")
64
-
if not name: # bare name, e.g. "gemma3-tools"
65
-
provider, name = "ollama", spec
66
-
if provider not in PROVIDERS:
67
-
sys.exit(f"unknown provider '{provider}' (known: {', '.join(PROVIDERS)})")
68
-
base, key_env = PROVIDERS[provider]
69
-
if key_env and not os.environ.get(key_env):
70
-
sys.exit(f"{key_env} is not set (needed for provider '{provider}')")
71
-
api_key = os.environ.get(key_env, "ollama") if key_env else "ollama"
72
-
return OpenAIChatModel(name, provider=OpenAIProvider(base_url=base, api_key=api_key))
73
-
74
-
75
-
# --- tasks: intentionally reference-dependent -----------------------------
76
-
# each fixes a signature (so the hidden oracle compiles), targets a langref
77
-
# feature weak models misremember, and ships a `reference` (selftest only).
78
-
79
-
80
-
@dataclass
81
-
class Task:
82
-
id: str
83
-
prompt: str
84
-
oracle: str
85
-
reference: str
86
-
note: str = ""
87
-
88
-
89
-
# every oracle forces RUNTIME inputs (`var x = ...; _ = &x;`) so `zig test` truly
90
-
# executes the agent's function at runtime rather than comptime-folding a literal.
91
-
TASKS: list[Task] = [
92
-
Task(
93
-
"intcast",
94
-
"Implement `pub fn toU8(x: u32) u8` that narrows x to a u8 using @intCast "
95
-
"(assume x fits). Make it compile and work.",
96
-
oracle='test "o" { const s=@import("std"); var x: u32 = 200; _ = &x; try s.testing.expectEqual(@as(u8, 200), toU8(x)); }',
97
-
reference="pub fn toU8(x: u32) u8 { return @intCast(x); }",
98
-
note="@intCast single-arg form (old: @intCast(u8, x))",
99
-
),
100
-
Task(
101
-
"for-index",
102
-
"Implement `pub fn indexOf(haystack: []const u8, needle: u8) ?usize` returning "
103
-
"the index of the first occurrence of needle, or null.",
104
-
oracle='test "o" { const s=@import("std"); var buf = [_]u8{\'a\',\'b\',\'c\'}; const h: []const u8 = &buf; var n: u8 = \'c\'; _ = &n; try s.testing.expectEqual(@as(?usize,2), indexOf(h,n)); var m: u8 = \'z\'; _ = &m; try s.testing.expectEqual(@as(?usize,null), indexOf(h,m)); }',
105
-
reference="pub fn indexOf(haystack: []const u8, needle: u8) ?usize { for (haystack, 0..) |c, i| { if (c == needle) return i; } return null; }",
106
-
note="for-loop index capture `for (xs, 0..) |x, i|`",
107
-
),
108
-
Task(
109
-
"switch-enum",
110
-
"Define `const Dir = enum { north, south, east, west };` and `pub fn dx(d: Dir) i8` "
111
-
"returning 1 for east, -1 for west, 0 otherwise, using a switch.",
112
-
oracle='test "o" { const s=@import("std"); var d: Dir = .east; _ = &d; try s.testing.expectEqual(@as(i8,1), dx(d)); d = .west; try s.testing.expectEqual(@as(i8,-1), dx(d)); d = .north; try s.testing.expectEqual(@as(i8,0), dx(d)); }',
113
-
reference="const Dir = enum { north, south, east, west }; pub fn dx(d: Dir) i8 { return switch (d) { .east => 1, .west => -1, .north, .south => 0 }; }",
114
-
note="exhaustive switch on enum",
115
-
),
116
-
Task(
117
-
"labeled-block",
118
-
"Implement `pub fn maxOf(a: i32, b: i32) i32` returning the larger value, computed "
119
-
"as a labeled block expression (`blk: { ... break :blk ...; }`).",
120
-
oracle='test "o" { const s=@import("std"); var a: i32 = 3; var b: i32 = 7; _ = &a; _ = &b; try s.testing.expectEqual(@as(i32,7), maxOf(a,b)); a = 9; b = 2; try s.testing.expectEqual(@as(i32,9), maxOf(a,b)); }',
121
-
reference="pub fn maxOf(a: i32, b: i32) i32 { return blk: { if (a > b) break :blk a; break :blk b; }; }",
122
-
note="labeled block + break :label value",
123
-
),
124
-
Task(
125
-
"error-union",
126
-
"Define `const E = error{ Bad };` and `pub fn addOk(a: E!i32, b: E!i32) E!i32` that "
127
-
"returns the sum, propagating either error with `try`.",
128
-
oracle='test "o" { const s=@import("std"); var a: i32 = 2; var b: i32 = 3; _ = &a; _ = &b; try s.testing.expectEqual(@as(i32,5), try addOk(a,b)); try s.testing.expectError(E.Bad, addOk(E.Bad,b)); }',
129
-
reference="const E = error{ Bad }; pub fn addOk(a: E!i32, b: E!i32) E!i32 { return (try a) + (try b); }",
130
-
note="error union + try propagation",
131
-
),
132
-
Task(
133
-
"sentinel-slice",
134
-
"Implement `pub fn slice3(arr: *const [3:0]u8) []const u8` returning the 3 elements "
135
-
"as a slice (excluding the sentinel).",
136
-
oracle='test "o" { const s=@import("std"); var a: [3:0]u8 = .{ \'a\', \'b\', \'c\' }; _ = &a; try s.testing.expectEqualSlices(u8, "abc", slice3(&a)); }',
137
-
reference="pub fn slice3(arr: *const [3:0]u8) []const u8 { return arr[0..3]; }",
138
-
note="sentinel-terminated array, slicing",
139
-
),
140
-
Task(
141
-
"cstrlen",
142
-
"Implement `pub fn cstrLen(s: [*:0]const u8) usize` returning the number of bytes "
143
-
"before the 0 sentinel (like C strlen).",
144
-
oracle='test "o" { const s=@import("std"); var buf = [_:0]u8{\'h\',\'e\',\'l\',\'l\',\'o\'}; _ = &buf; const p: [*:0]const u8 = &buf; try s.testing.expectEqual(@as(usize,5), cstrLen(p)); }',
145
-
reference="pub fn cstrLen(s: [*:0]const u8) usize { var i: usize = 0; while (s[i] != 0) i += 1; return i; }",
146
-
note="sentinel-terminated many-item pointer",
147
-
),
148
-
]
149
-
150
-
151
-
# --- objective oracle -----------------------------------------------------
152
-
153
-
154
-
def zig_test_source(zig_bin: str, source: str) -> tuple[bool, str]:
155
-
with tempfile.TemporaryDirectory() as d:
156
-
f = Path(d) / "t.zig"
157
-
f.write_text(source)
158
-
try:
159
-
r = subprocess.run([zig_bin, "test", str(f)], capture_output=True, text=True, timeout=60)
160
-
except FileNotFoundError:
161
-
sys.exit(f"zig not found ('{zig_bin}')")
162
-
except subprocess.TimeoutExpired:
163
-
return False, "timed out"
164
-
return r.returncode == 0, (r.stderr or r.stdout).strip()
165
-
166
-
167
-
def selftest(zig_bin: str) -> int:
168
-
fails = 0
169
-
for t in TASKS:
170
-
ok, detail = zig_test_source(zig_bin, t.reference + "\n" + t.oracle)
171
-
print(f" {'PASS' if ok else 'FAIL'} {t.id:<14} {t.note}")
172
-
if not ok:
173
-
print(f" oracle rejected reference: {detail.splitlines()[-1] if detail else ''}")
174
-
fails += 1
175
-
print(f"\n{len(TASKS) - fails}/{len(TASKS)} oracles valid")
176
-
return 1 if fails else 0
177
-
178
-
179
-
# --- eval agent -----------------------------------------------------------
180
-
181
-
182
-
@dataclass
183
-
class Step:
184
-
tool: str
185
-
args: dict
186
-
result: str
187
-
188
-
189
-
@dataclass
190
-
class Deps:
191
-
workdir: Path
192
-
zig_bin: str
193
-
zigman_bin: str
194
-
trace: list[Step] = field(default_factory=list)
195
-
last_file: str | None = None
196
-
197
-
198
-
# terse + imperative: a tutorial-style prompt makes weak models narrate tool calls
199
-
# instead of emitting them. the zigman arm just gets one extra line pointing at the
200
-
# tool — a lean nudge, not a heavy skill preamble (which destabilized formatting on
201
-
# weak local models). all tools are plain pydantic-ai function tools.
202
-
def eval_system(with_zigman: bool) -> str:
203
-
base = (
204
-
"You are an expert Zig programmer. Solve the task by USING the tools — never "
205
-
"describe what you would do, just call them. write_file saves Zig code (code "
206
-
"goes in the content argument). zig_test compiles and runs a file; read its "
207
-
"output and fix errors. "
208
-
)
209
-
nudge = (
210
-
"When unsure of exact Zig syntax, call zigman to read the official language "
211
-
"reference first (e.g. zigman('casting')). "
212
-
if with_zigman else ""
213
-
)
214
-
return base + nudge + "Start now."
215
-
216
-
217
-
def build_eval_agent(model_spec: str, with_zigman: bool) -> Agent[Deps]:
218
-
m = make_model(model_spec)
219
-
agent = Agent(m, deps_type=Deps, system_prompt=eval_system(with_zigman))
220
-
221
-
@agent.tool
222
-
async def write_file(ctx: RunContext[Deps], filename: str, content: str) -> str:
223
-
"""Write a Zig source file into the working directory. Put the full source in `content`."""
224
-
(ctx.deps.workdir / filename).write_text(content)
225
-
ctx.deps.last_file = filename
226
-
ctx.deps.trace.append(Step("write_file", {"filename": filename}, f"wrote {len(content)} bytes"))
227
-
return f"wrote {filename} ({len(content)} bytes)"
228
-
229
-
@agent.tool
230
-
async def zig_test(ctx: RunContext[Deps], filename: str) -> str:
231
-
"""Compile and run `zig test` on a file in the working directory; returns compiler output."""
232
-
try:
233
-
r = subprocess.run(
234
-
[ctx.deps.zig_bin, "test", str(ctx.deps.workdir / filename)],
235
-
capture_output=True, text=True, timeout=60, cwd=ctx.deps.workdir,
236
-
)
237
-
res = f"exit={r.returncode}\n{(r.stdout + r.stderr).strip()[:1500]}"
238
-
except Exception as e: # noqa: BLE001
239
-
res = f"error: {e}"
240
-
ctx.deps.trace.append(Step("zig_test", {"filename": filename}, res[:600]))
241
-
return res
242
-
243
-
if with_zigman:
244
-
245
-
@agent.tool
246
-
async def zigman(ctx: RunContext[Deps], query: str) -> str:
247
-
"""Read a section of the official Zig language reference (e.g. "comptime", "casting", "sentinel")."""
248
-
try:
249
-
r = subprocess.run(
250
-
[ctx.deps.zigman_bin, "--no-pager", query], capture_output=True, text=True, timeout=30
251
-
)
252
-
res = (r.stdout or r.stderr or "(no output)")[:4000]
253
-
except Exception as e: # noqa: BLE001
254
-
res = f"error: {e}"
255
-
ctx.deps.trace.append(Step("zigman", {"query": query}, res[:300]))
256
-
return res
257
-
258
-
return agent
259
-
260
-
261
-
async def run_arm(task: Task, args, with_zigman: bool) -> dict:
262
-
agent = build_eval_agent(args.agent_model, with_zigman)
263
-
with tempfile.TemporaryDirectory() as d:
264
-
deps = Deps(workdir=Path(d), zig_bin=args.zig, zigman_bin=args.zigman)
265
-
try:
266
-
res = await agent.run(task.prompt, deps=deps, usage_limits=UsageLimits(request_limit=10))
267
-
final = res.output or ""
268
-
except Exception as e: # noqa: BLE001
269
-
final = f"(agent error: {e})"
270
-
oracle_ok, oracle_detail = False, "no file written"
271
-
if deps.last_file:
272
-
code = (deps.workdir / deps.last_file).read_text()
273
-
oracle_ok, oracle_detail = zig_test_source(args.zig, code + "\n" + task.oracle)
274
-
return {
275
-
"oracle_ok": oracle_ok,
276
-
"oracle_detail": oracle_detail,
277
-
"used_zigman": any(s.tool == "zigman" for s in deps.trace),
278
-
"trace": [(s.tool, s.args, s.result) for s in deps.trace],
279
-
"final": final,
280
-
"trace_text": render_trace(task, deps, final, oracle_ok, oracle_detail),
281
-
}
282
-
283
-
284
-
def render_trace(task: Task, deps: Deps, final: str, oracle_ok: bool, oracle_detail: str) -> str:
285
-
lines = [f"TASK:\n{task.prompt}\n", "TOOL CALL TRACE:"]
286
-
if not deps.trace:
287
-
lines.append(" (the agent made no tool calls)")
288
-
for i, s in enumerate(deps.trace, 1):
289
-
lines.append(f" [{i}] {s.tool}({json.dumps(s.args)}) -> {s.result[:400]}")
290
-
lines.append(f"\nAGENT FINAL MESSAGE:\n{(final or '').strip()[:800]}")
291
-
lines.append(f"\nOBJECTIVE: hidden oracle test {'PASSED' if oracle_ok else 'FAILED'}")
292
-
if not oracle_ok:
293
-
lines.append(f" compiler said: {oracle_detail.splitlines()[-1] if oracle_detail else ''}")
294
-
return "\n".join(lines)
295
-
296
-
297
-
# --- optional llm judge ---------------------------------------------------
298
-
299
-
300
-
class Verdict(BaseModel):
301
-
passed: bool
302
-
reasoning: str
303
-
304
-
305
-
JUDGE_SYSTEM = (
306
-
"You judge whether a coding agent did a good job on a Zig task, given the task, its "
307
-
"tool-call trace, its final message, and the objective result of compiling its code "
308
-
"against a hidden test. A good job: the code solves the task and compiles & passes; "
309
-
"bonus for verifying with zig_test and consulting the reference when genuinely unsure. "
310
-
"Be strict: if the objective compile failed, it did not pass."
311
-
)
312
-
313
-
314
-
async def judge(args, trace_text: str, oracle_ok: bool) -> Verdict:
315
-
j = Agent(make_model(args.judge_model), output_type=Verdict, system_prompt=JUDGE_SYSTEM)
316
-
try:
317
-
return (await j.run(trace_text)).output
318
-
except Exception as e: # noqa: BLE001
319
-
return Verdict(passed=oracle_ok, reasoning=f"(judge error, fell back to oracle: {e})")
320
-
321
-
322
-
# --- driver ---------------------------------------------------------------
323
-
324
-
325
-
async def run_live(args) -> int:
326
-
tasks = [t for t in TASKS if not args.k or args.k in t.id]
327
-
arms = ["bare", "zigman"] if args.arms == "both" else [args.arms]
328
-
reports = Path(__file__).parent / "reports"
329
-
reports.mkdir(exist_ok=True)
330
-
results: dict[str, dict] = {}
331
-
332
-
for t in tasks:
333
-
results[t.id] = {}
334
-
for arm in arms:
335
-
with_zigman = arm == "zigman"
336
-
# average over trials (weak models are noisy); record best-effort detail
337
-
passes = 0
338
-
used = 0
339
-
last = None
340
-
for _ in range(args.trials):
341
-
r = await run_arm(t, args, with_zigman)
342
-
passes += r["oracle_ok"]
343
-
used += r["used_zigman"]
344
-
last = r
345
-
results[t.id][arm] = {
346
-
"pass_rate": passes / args.trials,
347
-
"zigman_use_rate": used / args.trials,
348
-
"sample": last,
349
-
}
350
-
mark = f"{passes}/{args.trials}"
351
-
note = f" zigman-used {used}/{args.trials}" if with_zigman else ""
352
-
print(f" {arm:<7} {t.id:<14} oracle {mark}{note}")
353
-
if with_zigman and args.judge:
354
-
v = await judge(args, last["trace_text"], last["oracle_ok"])
355
-
results[t.id][arm]["judge"] = {"passed": v.passed, "reasoning": v.reasoning}
356
-
print(f" judge: {'PASS' if v.passed else 'FAIL'} — {v.reasoning[:90]}")
357
-
358
-
# summary table + lift
359
-
print(f"\nagent: {args.agent_model} judge: {args.judge_model} (trials={args.trials})")
360
-
header = "task".ljust(16) + "".join(a.ljust(10) for a in arms)
361
-
print(header)
362
-
tot = {a: 0.0 for a in arms}
363
-
for t in tasks:
364
-
row = t.id.ljust(16)
365
-
for a in arms:
366
-
pr = results[t.id][a]["pass_rate"]
367
-
tot[a] += pr
368
-
row += f"{pr:.2f}".ljust(10)
369
-
print(row)
370
-
print("-" * len(header))
371
-
print("avg".ljust(16) + "".join(f"{tot[a] / len(tasks):.2f}".ljust(10) for a in arms))
372
-
if "bare" in arms and "zigman" in arms:
373
-
lift = (tot["zigman"] - tot["bare"]) / len(tasks)
374
-
print(f"\nzigman lift: {lift:+.2f} avg pass-rate")
375
-
gaps = [t.id for t in tasks if results[t.id]["bare"]["pass_rate"] < 1.0]
376
-
print(f"reference-dependent (bare not perfect): {gaps or 'none — tasks too easy'}")
377
-
378
-
out = reports / f"agentic_{args.agent_model.replace('/', '_').replace(':', '_')}.json"
379
-
out.write_text(json.dumps(results, indent=2, default=str))
380
-
print(f"evidence: {out}")
381
-
return 0
382
-
383
-
384
-
def main() -> int:
385
-
p = argparse.ArgumentParser(description=__doc__)
386
-
sub = p.add_subparsers(dest="cmd", required=True)
387
-
388
-
st = sub.add_parser("selftest")
389
-
st.add_argument("--zig", default="zig")
390
-
391
-
rn = sub.add_parser("run")
392
-
rn.add_argument("--agent-model", dest="agent_model", default="ollama/gemma3-tools",
393
-
help="model under test, as provider/model (bare name => ollama). "
394
-
f"providers: {', '.join(PROVIDERS)}")
395
-
rn.add_argument("--judge-model", dest="judge_model", default=None,
396
-
help="defaults to --agent-model; pair a dumb agent with a smart judge")
397
-
rn.add_argument("--arms", choices=["both", "bare", "zigman"], default="both")
398
-
rn.add_argument("--trials", type=int, default=1, help="runs per (task, arm); weak models are noisy")
399
-
rn.add_argument("--judge", action="store_true", help="also run the llm judge on the zigman arm")
400
-
rn.add_argument("--zig", default="zig")
401
-
rn.add_argument("--zigman", default="zigman")
402
-
rn.add_argument("-k", default="")
403
-
404
-
args = p.parse_args()
405
-
if args.cmd == "selftest":
406
-
return selftest(args.zig)
407
-
if not args.judge_model:
408
-
args.judge_model = args.agent_model
409
-
return asyncio.run(run_live(args))
410
-
411
-
412
-
if __name__ == "__main__":
413
-
raise SystemExit(main())
-219
evals/opencode_runner.py
-219
evals/opencode_runner.py
···
1
-
# /// script
2
-
# requires-python = ">=3.10"
3
-
# dependencies = ["pydantic-ai>=1.0", "pydantic-ai-skills>=0.8.0"]
4
-
# ///
5
-
"""run the zig eval through opencode as a real headless coding agent.
6
-
7
-
opencode drives the model with its own robust file/bash tools (so code flows
8
-
through tool calls, not fenced prose) and native skill/instruction delivery. we
9
-
parse opencode's JSON events for the tool-call trace, then score the produced
10
-
file with the same `zig test` oracle and (optionally) the same llm judge as the
11
-
pydantic-ai harness.
12
-
13
-
ablation: the zigman arm drops a project AGENTS.md telling the agent to consult
14
-
`zigman` (on PATH) for the language reference; the bare arm doesn't. the model is
15
-
model-agnostic via opencode's `-m provider/model`.
16
-
17
-
usage:
18
-
uv run evals/opencode_runner.py --model ollama/llama3.1:8b
19
-
uv run evals/opencode_runner.py --model ollama/llama3.1:8b -k intcast --judge
20
-
"""
21
-
22
-
from __future__ import annotations
23
-
24
-
import argparse
25
-
import asyncio
26
-
import json
27
-
import os
28
-
import subprocess
29
-
import sys
30
-
import tempfile
31
-
from pathlib import Path
32
-
33
-
sys.path.insert(0, str(Path(__file__).resolve().parent))
34
-
from eval import TASKS, JUDGE_SYSTEM, Verdict, make_model, zig_test_source # noqa: E402
35
-
from pydantic_ai import Agent # noqa: E402
36
-
37
-
REPO = Path(__file__).resolve().parent.parent
38
-
ZIGMAN_BIN_DIR = str(Path.home() / ".local/bin") # where install.sh puts zigman
39
-
40
-
OPENCODE = str(Path.home() / ".opencode/bin/opencode")
41
-
42
-
AGENTS_MD = """# project guidance
43
-
44
-
You are writing Zig. Zig syntax is precise and version-sensitive. When you are
45
-
unsure of exact syntax or semantics, run the `zigman` CLI to read the official
46
-
Zig language reference BEFORE writing — e.g. `zigman casting`, `zigman for`,
47
-
`zigman switch`, `zigman sentinel`, `zigman "Error Union Type"`. `zigman` is on
48
-
your PATH. Prefer reading the reference over guessing.
49
-
"""
50
-
51
-
OPENCODE_CONFIG = {
52
-
"$schema": "https://opencode.ai/config.json",
53
-
"permission": {"edit": "allow", "bash": "allow", "webfetch": "deny"},
54
-
"provider": {
55
-
"ollama": {
56
-
"npm": "@ai-sdk/openai-compatible",
57
-
"name": "Ollama local",
58
-
"options": {"baseURL": os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434/v1")},
59
-
"models": {"qwen2.5-coder:7b": {}, "llama3.1:8b": {}, "gemma3-tools": {}, "gemma3:12b": {}},
60
-
}
61
-
},
62
-
}
63
-
64
-
65
-
def parse_events(path: Path) -> dict:
66
-
"""extract tool-call trace + zigman usage from opencode's JSON event stream."""
67
-
trace, zigman_queries = [], []
68
-
for line in path.read_text().splitlines() if path.exists() else []:
69
-
line = line.strip()
70
-
if not line:
71
-
continue
72
-
try:
73
-
e = json.loads(line)
74
-
except json.JSONDecodeError:
75
-
continue
76
-
if e.get("type") != "tool_use":
77
-
continue
78
-
p = e.get("part", {})
79
-
tool = p.get("tool")
80
-
st = p.get("state", {})
81
-
inp = st.get("input", {}) or {}
82
-
trace.append({"tool": tool, "status": st.get("status"), "input": _short(inp)})
83
-
if tool == "bash":
84
-
cmd = inp.get("command", "")
85
-
if "zigman" in cmd:
86
-
zigman_queries.append(cmd.strip()[:120])
87
-
return {"trace": trace, "zigman_queries": zigman_queries}
88
-
89
-
90
-
def _short(inp: dict) -> dict:
91
-
out = {}
92
-
for k, v in inp.items():
93
-
s = str(v)
94
-
out[k] = s if len(s) <= 200 else s[:200] + "…"
95
-
return out
96
-
97
-
98
-
def run_opencode(task, model_spec: str, with_zigman: bool, timeout_s: int) -> dict:
99
-
d = Path(tempfile.mkdtemp(prefix="oc-eval-"))
100
-
(d / "opencode.json").write_text(json.dumps(OPENCODE_CONFIG, indent=2))
101
-
if with_zigman:
102
-
(d / "AGENTS.md").write_text(AGENTS_MD)
103
-
prompt = task.prompt + " Write your complete solution to a file named solution.zig in the current directory."
104
-
105
-
env = dict(os.environ)
106
-
env["PATH"] = ZIGMAN_BIN_DIR + os.pathsep + env.get("PATH", "")
107
-
events = d / "events.jsonl"
108
-
timed_out = False
109
-
with events.open("w") as fh:
110
-
try:
111
-
subprocess.run(
112
-
[OPENCODE, "run", "--dir", str(d), "-m", model_spec, "--format", "json", prompt],
113
-
cwd=d, env=env, stdout=fh, stderr=subprocess.DEVNULL, timeout=timeout_s,
114
-
)
115
-
except subprocess.TimeoutExpired:
116
-
timed_out = True
117
-
118
-
parsed = parse_events(events)
119
-
sol = d / "solution.zig"
120
-
if not sol.exists():
121
-
cands = list(d.glob("*.zig"))
122
-
sol = cands[0] if cands else None
123
-
oracle_ok, oracle_detail = False, "no .zig file produced"
124
-
code = ""
125
-
if sol and sol.exists():
126
-
code = sol.read_text()
127
-
oracle_ok, oracle_detail = zig_test_source("zig", code + "\n" + task.oracle)
128
-
return {
129
-
"task": task.id,
130
-
"timed_out": timed_out,
131
-
"oracle_ok": oracle_ok,
132
-
"oracle_detail": oracle_detail,
133
-
"used_zigman": bool(parsed["zigman_queries"]),
134
-
"zigman_queries": parsed["zigman_queries"],
135
-
"n_tool_calls": len(parsed["trace"]),
136
-
"trace": parsed["trace"],
137
-
"code": code,
138
-
"workdir": str(d),
139
-
}
140
-
141
-
142
-
def render_for_judge(task, r: dict) -> str:
143
-
lines = [f"TASK:\n{task.prompt}\n", "TOOL CALL TRACE:"]
144
-
for i, s in enumerate(r["trace"], 1):
145
-
lines.append(f" [{i}] {s['tool']} {s['status']} {json.dumps(s['input'])[:200]}")
146
-
if r["zigman_queries"]:
147
-
lines.append(f"\nZIGMAN QUERIES: {r['zigman_queries']}")
148
-
lines.append(f"\nPRODUCED CODE:\n{r['code'][:800]}")
149
-
lines.append(f"\nOBJECTIVE: hidden oracle test {'PASSED' if r['oracle_ok'] else 'FAILED'}")
150
-
if not r["oracle_ok"]:
151
-
lines.append(f" compiler: {(r['oracle_detail'].splitlines() or [''])[-1][:120]}")
152
-
return "\n".join(lines)
153
-
154
-
155
-
async def main_async(args) -> int:
156
-
tasks = [t for t in TASKS if not args.k or args.k in t.id]
157
-
arms = ["bare", "zigman"] if args.arms == "both" else [args.arms]
158
-
reports = Path(__file__).parent / "reports"
159
-
reports.mkdir(exist_ok=True)
160
-
results: dict[str, dict] = {}
161
-
162
-
for t in tasks:
163
-
results[t.id] = {}
164
-
for arm in arms:
165
-
r = await asyncio.to_thread(
166
-
run_opencode, t, args.model, arm == "zigman", args.timeout
167
-
)
168
-
results[t.id][arm] = r
169
-
to = " TIMEOUT" if r["timed_out"] else ""
170
-
uz = f" zigman:{r['zigman_queries']}" if r["used_zigman"] else (" zigman:none" if arm == "zigman" else "")
171
-
print(f" {arm:<7} {t.id:<14} oracle {'PASS' if r['oracle_ok'] else 'fail'} "
172
-
f"({r['n_tool_calls']} calls{to}){uz}")
173
-
if arm == "zigman" and args.judge:
174
-
v = await Agent(make_model(args.model), output_type=Verdict, system_prompt=JUDGE_SYSTEM).run(
175
-
render_for_judge(t, r)
176
-
)
177
-
results[t.id][arm]["judge"] = {"passed": v.output.passed, "reasoning": v.output.reasoning}
178
-
print(f" judge: {'PASS' if v.output.passed else 'FAIL'} — {v.output.reasoning[:90]}")
179
-
180
-
print(f"\nrunner: opencode model: {args.model}")
181
-
header = "task".ljust(16) + "".join(a.ljust(8) for a in arms)
182
-
print(header)
183
-
tot = {a: 0 for a in arms}
184
-
for t in tasks:
185
-
row = t.id.ljust(16)
186
-
for a in arms:
187
-
ok = results[t.id][a]["oracle_ok"]
188
-
tot[a] += ok
189
-
row += ("pass" if ok else "·").ljust(8)
190
-
print(row)
191
-
print("-" * len(header))
192
-
print("total".ljust(16) + "".join(f"{tot[a]}/{len(tasks)}".ljust(8) for a in arms))
193
-
if "zigman" in arms:
194
-
used = sum(results[t.id]["zigman"]["used_zigman"] for t in tasks)
195
-
print(f"zigman actually used in {used}/{len(tasks)} zigman-arm runs")
196
-
if "bare" in arms and "zigman" in arms:
197
-
print(f"zigman lift: {tot['zigman'] - tot['bare']:+d}")
198
-
199
-
out = reports / f"opencode_{args.model.replace('/', '_').replace(':', '_')}.json"
200
-
out.write_text(json.dumps(results, indent=2, default=str))
201
-
print(f"evidence: {out}")
202
-
return 0
203
-
204
-
205
-
def main() -> int:
206
-
p = argparse.ArgumentParser(description=__doc__)
207
-
p.add_argument("--model", default="ollama/llama3.1:8b", help="opencode model: provider/model")
208
-
p.add_argument("--arms", choices=["both", "bare", "zigman"], default="both")
209
-
p.add_argument("--judge", action="store_true")
210
-
p.add_argument("--timeout", type=int, default=240, help="hard per-run timeout (s)")
211
-
p.add_argument("-k", default="")
212
-
args = p.parse_args()
213
-
if not Path(OPENCODE).exists():
214
-
sys.exit(f"opencode not found at {OPENCODE}")
215
-
return asyncio.run(main_async(args))
216
-
217
-
218
-
if __name__ == "__main__":
219
-
raise SystemExit(main())
-99
evals/prove.py
-99
evals/prove.py
···
1
-
# /// script
2
-
# requires-python = ">=3.10"
3
-
# dependencies = ["pydantic-ai>=1.0", "pydantic-ai-skills>=0.8.0"]
4
-
# ///
5
-
"""prove gemma (a) loads the zigman skill and (b) calls the zigman tool.
6
-
7
-
skills are delivered the phi way: pydantic_ai_skills.SkillsToolset over the
8
-
agentskills.io `skills/` dir (auto preamble + `load_skill` for the body). the
9
-
zigman CLI is a plain exec tool the skill points at. we dump every tool call.
10
-
11
-
usage: uv run evals/prove.py [--model gemma3-tools]
12
-
"""
13
-
14
-
from __future__ import annotations
15
-
16
-
import argparse
17
-
import subprocess
18
-
from pathlib import Path
19
-
20
-
from pydantic_ai import Agent
21
-
from pydantic_ai.messages import ToolCallPart
22
-
from pydantic_ai.models.openai import OpenAIChatModel
23
-
from pydantic_ai.providers.openai import OpenAIProvider
24
-
from pydantic_ai_skills import SkillsToolset
25
-
26
-
REPO = Path(__file__).resolve().parent.parent
27
-
SKILLS_DIR = REPO / "skills"
28
-
29
-
# NOTE: no markdown-fence instructions anywhere. with the gemma3-tools Modelfile,
30
-
# tool calls must be emitted as a BARE json array; priming the model to use ```
31
-
# fences (e.g. "output a ```zig block") makes it fence the tool call too, and
32
-
# ollama's parser then ignores it. keep the call phase fence-free.
33
-
SYSTEM = (
34
-
"You are an expert Zig programmer. You have skills available; consult them. "
35
-
"Use the zigman tool to read the official Zig language reference whenever you "
36
-
"need precise syntax."
37
-
)
38
-
39
-
PROMPT = (
40
-
"What does the Zig language reference say about sentinel-terminated pointers? "
41
-
"Use the zigman tool to look it up."
42
-
)
43
-
44
-
45
-
def main() -> int:
46
-
ap = argparse.ArgumentParser()
47
-
ap.add_argument("--model", default="gemma3-tools")
48
-
ap.add_argument("--base-url", default="http://localhost:11434/v1")
49
-
ap.add_argument("--api-key", default="ollama")
50
-
ap.add_argument("--no-skills", action="store_true", help="drop SkillsToolset to isolate format breakage")
51
-
args = ap.parse_args()
52
-
53
-
model = OpenAIChatModel(args.model, provider=OpenAIProvider(base_url=args.base_url, api_key=args.api_key))
54
-
toolsets = []
55
-
if not args.no_skills:
56
-
toolsets.append(SkillsToolset(directories=[SKILLS_DIR], exclude_tools=["run_skill_script"]))
57
-
agent = Agent(model, toolsets=toolsets, system_prompt=SYSTEM)
58
-
59
-
@agent.tool_plain
60
-
def zigman(query: str) -> str:
61
-
"""Read a section of the official Zig language reference.
62
-
63
-
query: a topic like "sentinel", "slices", "comptime", "errors".
64
-
"""
65
-
r = subprocess.run(
66
-
["zigman", "--no-pager", query], capture_output=True, text=True, timeout=30
67
-
)
68
-
return (r.stdout or r.stderr or "(no output)")[:4000]
69
-
70
-
import asyncio
71
-
72
-
async def go():
73
-
return await agent.run(PROMPT)
74
-
75
-
res = asyncio.run(go())
76
-
77
-
print("=== tool calls gemma made ===")
78
-
n = 0
79
-
for m in res.all_messages():
80
-
for part in getattr(m, "parts", []):
81
-
if isinstance(part, ToolCallPart):
82
-
n += 1
83
-
args_str = str(part.args)[:160]
84
-
print(f" [{n}] {part.tool_name}({args_str})")
85
-
print(f"\ntotal tool calls: {n}")
86
-
zigman_calls = sum(
87
-
1
88
-
for m in res.all_messages()
89
-
for part in getattr(m, "parts", [])
90
-
if isinstance(part, ToolCallPart) and part.tool_name == "zigman"
91
-
)
92
-
print(f"zigman tool calls: {zigman_calls}")
93
-
print("\n=== final answer (first 600 chars) ===")
94
-
print((res.output or "")[:600])
95
-
return 0 if zigman_calls > 0 else 1
96
-
97
-
98
-
if __name__ == "__main__":
99
-
raise SystemExit(main())
+271
evals/run.py
+271
evals/run.py
···
1
+
#!/usr/bin/env python3
2
+
"""zigman eval loop: drive a local coding agent (Pi) over real zig tasks.
3
+
4
+
stack (all local, no GUI): Pi (@earendil-works/pi-coding-agent) as the agent
5
+
harness, a gemma-4 model served OpenAI-compatible by mlx_vlm.server, the
6
+
`zig test` compiler as the objective oracle. Pi runs headless (`-p --mode json`),
7
+
so we parse its event stream for the tool-call trace.
8
+
9
+
ablation: the zigman arm loads skills/zigman via Pi's native `--skill` and has
10
+
the `zigman` CLI on PATH; the bare arm doesn't. the question: does a competent
11
+
local agent write better zig with zigman than without?
12
+
13
+
no python deps — this just orchestrates the `pi` and `zig` CLIs.
14
+
15
+
usage:
16
+
python3 evals/run.py selftest # validate oracles, no model
17
+
python3 evals/run.py run # ablate over the corpus
18
+
python3 evals/run.py run -k intcast --arms zigman
19
+
python3 evals/run.py run --model <id> --provider local
20
+
21
+
start the model server first (or point --base-url at LM Studio on the same port):
22
+
mlx_vlm.server --model ~/models/gemma-4-12B-it-8bit --port 1234
23
+
"""
24
+
25
+
from __future__ import annotations
26
+
27
+
import argparse
28
+
import json
29
+
import os
30
+
import shutil
31
+
import subprocess
32
+
import sys
33
+
import tempfile
34
+
import urllib.request
35
+
from dataclasses import dataclass
36
+
from pathlib import Path
37
+
38
+
REPO = Path(__file__).resolve().parent.parent
39
+
SKILL = REPO / "skills" / "zigman"
40
+
ZIGMAN_BIN_DIR = str(Path.home() / ".local/bin")
41
+
DEFAULT_MODEL = str(Path.home() / "models/gemma-4-12B-it-8bit")
42
+
DEFAULT_BASE_URL = "http://localhost:1234/v1"
43
+
44
+
45
+
# --- tasks: intentionally reference-dependent (langref-documented features) ---
46
+
47
+
48
+
@dataclass
49
+
class Task:
50
+
id: str
51
+
prompt: str
52
+
oracle: str # hidden `test {...}` over the required decl, forces RUNTIME execution
53
+
reference: str # known-good solution, selftest only
54
+
note: str = ""
55
+
56
+
57
+
TASKS: list[Task] = [
58
+
Task("intcast",
59
+
"Implement `pub fn toU8(x: u32) u8` that narrows x to a u8 using @intCast (assume x fits).",
60
+
'test "o" { const s=@import("std"); var x: u32 = 200; _ = &x; try s.testing.expectEqual(@as(u8, 200), toU8(x)); }',
61
+
"pub fn toU8(x: u32) u8 { return @intCast(x); }",
62
+
"@intCast single-arg form (old: @intCast(u8, x))"),
63
+
Task("for-index",
64
+
"Implement `pub fn indexOf(haystack: []const u8, needle: u8) ?usize` returning the index of the first occurrence of needle, or null.",
65
+
'test "o" { const s=@import("std"); var buf=[_]u8{\'a\',\'b\',\'c\'}; const h: []const u8 = &buf; var n: u8=\'c\'; _=&n; try s.testing.expectEqual(@as(?usize,2), indexOf(h,n)); var m: u8=\'z\'; _=&m; try s.testing.expectEqual(@as(?usize,null), indexOf(h,m)); }',
66
+
"pub fn indexOf(haystack: []const u8, needle: u8) ?usize { for (haystack, 0..) |c, i| { if (c == needle) return i; } return null; }",
67
+
"for-loop index capture `for (xs, 0..) |x, i|`"),
68
+
Task("switch-enum",
69
+
"Define `const Dir = enum { north, south, east, west };` and `pub fn dx(d: Dir) i8` returning 1 for east, -1 for west, 0 otherwise, using a switch.",
70
+
'test "o" { const s=@import("std"); var d: Dir = .east; _=&d; try s.testing.expectEqual(@as(i8,1), dx(d)); d=.west; try s.testing.expectEqual(@as(i8,-1), dx(d)); d=.north; try s.testing.expectEqual(@as(i8,0), dx(d)); }',
71
+
"const Dir = enum { north, south, east, west }; pub fn dx(d: Dir) i8 { return switch (d) { .east => 1, .west => -1, .north, .south => 0 }; }",
72
+
"exhaustive switch on enum"),
73
+
Task("labeled-block",
74
+
"Implement `pub fn maxOf(a: i32, b: i32) i32` returning the larger value, computed as a labeled block expression (`blk: { ... break :blk ...; }`).",
75
+
'test "o" { const s=@import("std"); var a: i32=3; var b: i32=7; _=&a; _=&b; try s.testing.expectEqual(@as(i32,7), maxOf(a,b)); a=9; b=2; try s.testing.expectEqual(@as(i32,9), maxOf(a,b)); }',
76
+
"pub fn maxOf(a: i32, b: i32) i32 { return blk: { if (a > b) break :blk a; break :blk b; }; }",
77
+
"labeled block + break :label value"),
78
+
Task("error-union",
79
+
"Define `const E = error{ Bad };` and `pub fn addOk(a: E!i32, b: E!i32) E!i32` that returns the sum, propagating either error with `try`.",
80
+
'test "o" { const s=@import("std"); var a: i32=2; var b: i32=3; _=&a; _=&b; try s.testing.expectEqual(@as(i32,5), try addOk(a,b)); try s.testing.expectError(E.Bad, addOk(E.Bad,b)); }',
81
+
"const E = error{ Bad }; pub fn addOk(a: E!i32, b: E!i32) E!i32 { return (try a) + (try b); }",
82
+
"error union + try propagation"),
83
+
Task("sentinel-slice",
84
+
"Implement `pub fn slice3(arr: *const [3:0]u8) []const u8` returning the 3 elements as a slice (excluding the sentinel).",
85
+
'test "o" { const s=@import("std"); var a: [3:0]u8 = .{ \'a\', \'b\', \'c\' }; _=&a; try s.testing.expectEqualSlices(u8, "abc", slice3(&a)); }',
86
+
"pub fn slice3(arr: *const [3:0]u8) []const u8 { return arr[0..3]; }",
87
+
"sentinel-terminated array, slicing"),
88
+
Task("cstrlen",
89
+
"Implement `pub fn cstrLen(s: [*:0]const u8) usize` returning the number of bytes before the 0 sentinel (like C strlen).",
90
+
'test "o" { const s=@import("std"); var buf=[_:0]u8{\'h\',\'e\',\'l\',\'l\',\'o\'}; _=&buf; const p: [*:0]const u8 = &buf; try s.testing.expectEqual(@as(usize,5), cstrLen(p)); }',
91
+
"pub fn cstrLen(s: [*:0]const u8) usize { var i: usize = 0; while (s[i] != 0) i += 1; return i; }",
92
+
"sentinel-terminated many-item pointer"),
93
+
]
94
+
95
+
96
+
# --- objective oracle: zig test EXECUTES the code (not just compiles) ---------
97
+
98
+
99
+
def zig_test_source(source: str, zig: str = "zig") -> tuple[bool, str]:
100
+
with tempfile.TemporaryDirectory() as d:
101
+
f = Path(d) / "t.zig"
102
+
f.write_text(source)
103
+
try:
104
+
r = subprocess.run([zig, "test", str(f)], capture_output=True, text=True, timeout=60)
105
+
except FileNotFoundError:
106
+
sys.exit(f"zig not found ('{zig}')")
107
+
except subprocess.TimeoutExpired:
108
+
return False, "timed out"
109
+
return r.returncode == 0, (r.stderr or r.stdout).strip()
110
+
111
+
112
+
def selftest() -> int:
113
+
fails = 0
114
+
for t in TASKS:
115
+
ok, detail = zig_test_source(t.reference + "\n" + t.oracle)
116
+
print(f" {'PASS' if ok else 'FAIL'} {t.id:<14} {t.note}")
117
+
if not ok:
118
+
print(f" oracle rejected reference: {(detail.splitlines() or [''])[-1]}")
119
+
fails += 1
120
+
print(f"\n{len(TASKS) - fails}/{len(TASKS)} oracles valid")
121
+
return 1 if fails else 0
122
+
123
+
124
+
# --- Pi runner ----------------------------------------------------------------
125
+
126
+
127
+
def pi_bin() -> str:
128
+
return shutil.which("pi") or str(Path("/opt/homebrew/bin/pi"))
129
+
130
+
131
+
def server_up(base_url: str) -> bool:
132
+
try:
133
+
urllib.request.urlopen(base_url.rstrip("/") + "/models", timeout=3)
134
+
return True
135
+
except Exception: # noqa: BLE001
136
+
return False
137
+
138
+
139
+
def parse_events(stdout: str) -> dict:
140
+
"""extract completed tool calls from Pi's --mode json event stream."""
141
+
calls = {} # id -> {name, arguments}
142
+
for line in stdout.splitlines():
143
+
line = line.strip()
144
+
if not line:
145
+
continue
146
+
try:
147
+
e = json.loads(line)
148
+
except json.JSONDecodeError:
149
+
continue
150
+
msg = e.get("message") or {}
151
+
if e.get("type") == "message_end" and msg.get("role") == "assistant":
152
+
for part in msg.get("content", []):
153
+
if part.get("type") == "toolCall" and part.get("id"):
154
+
calls[part["id"]] = {"name": part.get("name"), "arguments": part.get("arguments", {})}
155
+
tools = list(calls.values())
156
+
zigman_qs = [
157
+
c["arguments"].get("command", "")
158
+
for c in tools
159
+
if c["name"] == "bash" and "zigman" in str(c["arguments"].get("command", ""))
160
+
]
161
+
return {"tools": tools, "zigman_queries": zigman_qs}
162
+
163
+
164
+
def run_pi(task: Task, args, with_zigman: bool) -> dict:
165
+
d = Path(tempfile.mkdtemp(prefix="pi-eval-"))
166
+
prompt = (
167
+
task.prompt
168
+
+ " Write your complete solution to a file named solution.zig in the current "
169
+
"directory, then verify it compiles by running `zig test solution.zig`."
170
+
)
171
+
cmd = [
172
+
pi_bin(), "-p", "--mode", "json", "--no-session",
173
+
"--provider", args.provider, "--model", args.model,
174
+
]
175
+
cmd += ["--skill", str(SKILL)] if with_zigman else ["--no-skills"]
176
+
cmd += [prompt]
177
+
178
+
env = dict(os.environ)
179
+
env["PATH"] = ZIGMAN_BIN_DIR + os.pathsep + env.get("PATH", "")
180
+
timed_out = False
181
+
try:
182
+
r = subprocess.run(cmd, cwd=d, env=env, capture_output=True, text=True, timeout=args.timeout)
183
+
stdout = r.stdout
184
+
except subprocess.TimeoutExpired as e:
185
+
stdout = e.stdout.decode() if isinstance(e.stdout, bytes) else (e.stdout or "")
186
+
timed_out = True
187
+
188
+
parsed = parse_events(stdout)
189
+
sol = d / "solution.zig"
190
+
if not sol.exists():
191
+
cands = sorted(d.glob("*.zig"))
192
+
sol = cands[0] if cands else None
193
+
oracle_ok, detail = False, "no .zig file produced"
194
+
code = ""
195
+
if sol and sol.exists():
196
+
code = sol.read_text()
197
+
oracle_ok, detail = zig_test_source(code + "\n" + task.oracle)
198
+
shutil.rmtree(d, ignore_errors=True)
199
+
return {
200
+
"oracle_ok": oracle_ok, "detail": detail, "timed_out": timed_out,
201
+
"used_zigman": bool(parsed["zigman_queries"]),
202
+
"zigman_queries": parsed["zigman_queries"],
203
+
"tools": [t["name"] for t in parsed["tools"]],
204
+
"code": code,
205
+
}
206
+
207
+
208
+
def run_loop(args) -> int:
209
+
if not server_up(args.base_url):
210
+
sys.exit(
211
+
f"model server not reachable at {args.base_url}. start one, e.g.:\n"
212
+
f" mlx_vlm.server --model {args.model} --port 1234\n"
213
+
f"(or point --base-url at LM Studio on the same port)"
214
+
)
215
+
tasks = [t for t in TASKS if not args.k or args.k in t.id]
216
+
arms = ["bare", "zigman"] if args.arms == "both" else [args.arms]
217
+
results: dict[str, dict] = {}
218
+
for t in tasks:
219
+
results[t.id] = {}
220
+
for arm in arms:
221
+
r = run_pi(t, args, with_zigman=(arm == "zigman"))
222
+
results[t.id][arm] = r
223
+
to = " TIMEOUT" if r["timed_out"] else ""
224
+
uz = (" zigman:" + (str(r["zigman_queries"]) if r["used_zigman"] else "none")) if arm == "zigman" else ""
225
+
print(f" {arm:<7} {t.id:<14} {'PASS' if r['oracle_ok'] else 'fail'} "
226
+
f"[{','.join(r['tools']) or 'no-tools'}{to}]{uz}")
227
+
228
+
print(f"\nagent: Pi model: {args.model}")
229
+
header = "task".ljust(16) + "".join(a.ljust(8) for a in arms)
230
+
print(header)
231
+
tot = {a: 0 for a in arms}
232
+
for t in tasks:
233
+
row = t.id.ljust(16)
234
+
for a in arms:
235
+
ok = results[t.id][a]["oracle_ok"]
236
+
tot[a] += ok
237
+
row += ("pass" if ok else "·").ljust(8)
238
+
print(row)
239
+
print("-" * len(header))
240
+
print("total".ljust(16) + "".join(f"{tot[a]}/{len(tasks)}".ljust(8) for a in arms))
241
+
if "zigman" in arms:
242
+
used = sum(results[t.id]["zigman"]["used_zigman"] for t in tasks)
243
+
print(f"zigman used in {used}/{len(tasks)} zigman-arm runs")
244
+
if "bare" in arms and "zigman" in arms:
245
+
print(f"zigman lift: {tot['zigman'] - tot['bare']:+d}")
246
+
247
+
reports = Path(__file__).parent / "reports"
248
+
reports.mkdir(exist_ok=True)
249
+
out = reports / f"pi_{Path(args.model).name}.json"
250
+
out.write_text(json.dumps(results, indent=2))
251
+
print(f"evidence: {out}")
252
+
return 0
253
+
254
+
255
+
def main() -> int:
256
+
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
257
+
sub = p.add_subparsers(dest="cmd", required=True)
258
+
sub.add_parser("selftest", help="validate oracles against reference solutions (no model)")
259
+
rn = sub.add_parser("run", help="ablate the corpus with Pi")
260
+
rn.add_argument("--provider", default="local", help="Pi provider name (from ~/.pi/agent/models.json)")
261
+
rn.add_argument("--model", default=DEFAULT_MODEL, help="model id for the provider")
262
+
rn.add_argument("--base-url", default=DEFAULT_BASE_URL, help="server endpoint (for the reachability check)")
263
+
rn.add_argument("--arms", choices=["both", "bare", "zigman"], default="both")
264
+
rn.add_argument("--timeout", type=int, default=300, help="hard per-run timeout (s)")
265
+
rn.add_argument("-k", default="", help="substring filter on task id")
266
+
args = p.parse_args()
267
+
return selftest() if args.cmd == "selftest" else run_loop(args)
268
+
269
+
270
+
if __name__ == "__main__":
271
+
raise SystemExit(main())