# zigman-eval

does zigman help a local coding agent write correct zig? an ablation (bare vs.
zigman) over a corpus of reference-dependent zig tasks, run through
[Pi](https://github.com/badlogic/pi-mono) (the agent) and scored by `zig test` (the
objective oracle — it *runs* the code, not just compiles it).

pure stdlib; it orchestrates the `pi` and `zig` CLIs as subprocesses.

## layout

```
src/zigman_eval/
  tasks.py    # the corpus + the compiler oracle (zig_test_source, grade, validate_oracles)
  config.py   # Config + resolution (model via arg/env, zigman via PATH, skill by walking up)
  pi.py       # run one task through Pi headless; parse its JSON events -> ArmResult
  loop.py     # ablate() -> Report; render() the table; save() the json
  cli.py      # `zigman-eval selftest | run`
tests/        # the oracles must accept their reference solutions
```

## use

first serve a tool-calling-capable local model (see the project's
`local-models/` notes — gemma-4 via `mlx_vlm.server` works):

```sh
mlx_vlm.server --model ~/models/gemma-4-12B-it-8bit --port 1234
```

then, from this directory:

```sh
uv run zigman-eval selftest                       # validate oracles, no model
uv run zigman-eval run --model <provider-model-id>   # ablate the corpus
uv run zigman-eval run --model <id> -k intcast --arms zigman
ZIGMAN_EVAL_MODEL=<id> uv run zigman-eval run     # model via env
uv run pytest                                     # the oracle tests
```

the `--model` is whatever your Pi provider calls the model (for the `local` provider
configured in `~/.pi/agent/models.json`, that's typically the model path).
