# evaluation axes for zigman

the quality loop scores zigman on every dimension below. each axis says what
*good* looks like and how it's judged: **objective** (a check or measurement we
run), **experiential** (signal from a weak model actually using zigman to do a
zig task — its friction is the data), or both. the constraint throughout: stay
within the existing command surface — improve sensibility, not scope.

## 1. API sensibility
intuitive, guessable invocation; minimal surprise; one obvious way.
- experiential: can the model pick the right `zigman <query>` without being told the
  exact anchor? does it reach for flags that don't exist?
- objective: flag/verb count, naming consistency, help legibility.

## 2. discoverability
you can find what you need without already knowing the section name.
- experiential: when a query is ambiguous, does the candidate list let the model
  recover and re-query correctly? does it ever give up?
- objective: `-l`, fuzzy resolution, search hit quality, completion coverage.

## 3. output quality
the rendered section is correct, complete, and *useful for writing code*.
- experiential: after reading a section, does the model's code compile? which
  sections led to passes vs. fails?
- objective: markdown fidelity (code fences, tables, inline code), no HTML cruft,
  faithful to the langref.

## 4. ergonomics & composability
plays well in pipes and agent loops.
- objective: data on stdout / diagnostics on stderr, exit codes, `--no-pager` and
  tty-detection, broken-pipe handling, deterministic non-interactive output.
- experiential: does tool output (truncation, length) fit a small model's context?

## 5. portability
runs anywhere with no surprises.
- objective: macos + linux × aarch64/x86_64 build & run; static linking; no runtime
  deps; honors `$XDG_CACHE_HOME`, `$PAGER`, `$BROWSER`; shell completions for
  zsh/bash/fish; sane install paths.

## 6. performance & footprint
fast to start, cheap to run, small to ship.
- objective: cold/warm latency, cache behaviour (TTL, refetch), binary size,
  per-call cost when used as a tool.

## 7. robustness & error handling
fails clearly, never confusingly.
- objective: bad query, offline-with/without-cache, invalid version, oversized
  input, missing `$HOME`. errors are actionable and on stderr.
- experiential: does a wrong guess produce a message the model can act on?

## 8. maintainability
easy to read, change, and trust.
- objective: lib/CLI split, test coverage (oracle suite + unit), file sizes,
  single-source docs (README → page), comment discipline.

## 9. aesthetics ("colors" / je ne sais quoi)
taste: help readability, naming, output formatting, the landing page, overall feel.
- objective: help layout, example quality, page scannability.
- experiential / human: does it feel good to use?

## 10. distribution & adoption
low-friction to install, update, and integrate.
- objective: `curl | sh` install, cross-client `SKILL.md`, source build, update path,
  no GitHub/registry dependency.

---

## how the loop uses these

1. **measure** (objective): `eval.py run` — the ablation. lift = zigman's value on
   weak-model zig correctness; per-task failures + the model's zigman queries are
   evidence for axes 1–4 and 7.
2. **critique** (experiential): `eval.py review` — the model does a task *using*
   zigman, then reflects per axis with one concrete, in-scope proposal each,
   grounded in what just happened.
3. **act**: apply the highest-value, in-surface proposals; re-run `run`; compare
   lift. repeat.

objective axes (5, 6, 8, 10) move via our own checks; experiential axes (1–4, 7, 9)
move via what the model's use reveals.
