does-it-tool#

Does this local model actually do tool calling? Point it at any OpenAI-compatible endpoint (ollama, vLLM, LM Studio, llama.cpp server) and it runs a small behavioural suite through pydantic-ai — real tool dispatch, no regex parsing — then scores what the model did.

It checks the things that actually break when you swap a frontier model for a small local one:

quickstart#

It's a pytest suite — one test per case, parametrized over the model you pass:

# any OpenAI-compatible endpoint works; ollama is the default
uv run pytest --model llama3.1:8b -v

# explicit endpoint
uv run pytest --model my-model --base-url http://localhost:8000/v1 --api-key sk-anything

# env vars instead of flags (handy in CI)
MODEL=gemma3-tools uv run pytest

Run a single case with the usual -k filter (-k dependent-chain). pytest's exit code gates a model swap in CI for free.

the catch with small models: the serving layer, not the model#

Most small open models can reason about tools fine. What breaks is the serving wrapper: it either doesn't expose tools at all, or doesn't parse the model's output back into structured tool_calls.

ollama is the canonical example. Its stock gemma3 template references .Tools zero times, so the API 400s every tool request before the model sees it. modelfiles/gemma3-tools.Modelfile fixes that — it renders the tool list into the opening turn and constrains the model to emit a bare JSON array (no markdown fences) that ollama's parser recognizes:

ollama create gemma3-tools -f modelfiles/gemma3-tools.Modelfile
uv run pytest --model gemma3-tools -v   # 6 passed

The same idea generalizes: if a model fails every case with a 400 or leaks raw JSON into the reply text, the fix is the template/parser, not the model. On vLLM, use the built-in per-model tool parser (--enable-auto-tool-choice --tool-call-parser ...) instead of a Modelfile.

extending the suite#

src/does_it_tool/suite.py is data. Add a tool in register_tools, append a Case with a validator over the recorded call sequence, done. Validators get the full list[Call] (name + args) so you can assert on arguments, ordering, and abstention — not just prose.

license#

MIT.