Weight-orthogonalization (abliteration) experiments on Qwen3.5-4B Soyuz LoRA using pass-vs-fail trajectory directions. 7-step pipeline, 8 model variants, full bench results.
- Python 57.2%
- Shell 42.8%
Aleksandr Nikolich
bfe0bfdc1fFinal phase2 results (4 of 5 experiments executed, exp3 deferred): - exp1 v9_multitok: tbench=1/17, HA20=5/20 - exp5 v7_agentonly: tbench=2/17, HA20=9/20, MMLU=2.24% - exp2 v6_hardpairs: tbench=2/17, HA20=8/20, MMLU=2.08% - exp4 v8_cfact: tbench=2/17, HA20=9/20 The central hypothesis was: dropping MMLU-Pi rows from the FAIL bucket (exp5 v7_agentonly, 0% MMLU-Pi share in both buckets) would preserve MMLU-Pro near baseline (58.72%). Instead MMLU collapsed to 2.24% — identical to the MMLU-heavy v2/v5_SVD variants. v6_hardpairs (same-task pairs, also MMLU-Pi-free) collapsed to 2.08%. Conclusion: the direction removed by mu_fail - mu_pass is not "MMLU knowledge" specifically; it is a more fundamental task-execution component that load-bears on MMLU even when MMLU-Pi sources are absent from the contrast. HA20 ceiling 9-10/20 is robust across every recipe tried (mean / SVD / Ridge / agent-only / within-task pairs / counterfactual injection). Three winners published to HF: - AlexWortega/qwen35-4b-soyuz-abliterated-v7_agentonly - AlexWortega/qwen35-4b-soyuz-abliterated-v6_hardpairs - AlexWortega/qwen35-4b-soyuz-abliterated-v8_cfact Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| data | ||
| docs | ||
| results | ||
| scripts | ||
| .gitignore | ||
| CLAUDE.md | ||
| README.md | ||
capability-vectors
Weight-orthogonalization ("abliteration") experiments on Qwen3.5-4B + Soyuz LoRA using pass-vs-fail directions extracted from the model's own evaluation traces.
See CLAUDE.md for the full recipe, results table, findings, and HuggingFace model links.
TL;DR
| Variant | tbench-17 | HermesAgent-20 | MMLU-Pro | EQ |
|---|---|---|---|---|
| baseline soyuz | 5/17 | 4/20 | 58.72% | 64.35 |
| abliterated-v2 (single-L mean, s=0.5) | 3/17 | 8/20 | 2.09% ⚠ | 51.10 |
| abliterated-v3-multi (per-layer ortho) | 2/17 | 6/20 | — | — |
| v5_SVD (top-1 SVD direction) | 1/17 | 10/20 | — | — |
| v5_REG (ridge reward-regression) | 1/17 | 10/20 | 2.06% ⚠ | — |
Method works for one capability axis (Hermes agent tool-use → 2.5×) but blasts general knowledge (MMLU-Pro → ~2%). Tradeoff comes from MMLU-Pi-agent fails being included in the contrast — work to fix is queued.
Layout
scripts/ is split per pipeline step (01_build_contrast → 07_bench). All intermediate artifacts (contrasts, activations, directions, models) live under /workspace/capvec_* on the training box; only the source code is in git.