octocode-eval - Claude MCP Skill

# Octocode Eval
Evaluate outcomes and improve with evidence, not vibes.
Flow: `ERROR-ANALYZE → FRAME(goal→KPI) → BASELINE → LOOP → JUDGE → CAPTURE → VERIFY → SUITE-EVOLVE`.
Modes: **ErrorAnalyze** · **Define** · **Run** · **Suite** · **Benchmark** · **Audit**.

## Lobby rules
- No goal→KPI link → STOP. No measurable primary → STOP.
- Narrative-only accept → REJECT. Editing harness/cases/graders to pass → REJECT.
- ACCEPT only if primary moves on held-out **and** guardrails hold.
- Prefer deterministic graders; binary/LLM next; humans calibrate. Grade outcomes over paths.
- **TDD for agents:** write or select a failing case / KPI check **before** mutating the subject; green only after the change (red → green → keep|discard).
- Public benches orient; private failure suites gate ships. Distrust saturated/contaminated boards.
- Freeze the harness during an experiment; evolve the suite only between experiments.

## Workflow
1. Error-analyze traces into a failure taxonomy; frame success, primary/leading metrics, guardrails, and decision rule.
2. Measure a fixed-budget baseline; make the smallest subject change; keep or discard from comparable results.
3. Judge grader quality, fairness, capability versus regression, and contamination; capture one durable lesson.
4. Verify held-out results and required checks; then add new failure cases between experiments.
Stop when goal/KPI is undefined, checks did not run, the harness changed to pass, or another loop cannot change the verdict.

## Smart routes — load only what the current step needs
- When deriving failures, load `references/error-analysis.md`; when connecting intent to measures load `references/goal-kpi-cascade.md`, then fill `references/kpi-contract.md` — make success and budget explicit.
- When choosing experiment, suite, or meta scope, load `references/nested-loops.md`; for the inner keep/discard cycle load `references/agent-loop.md` — avoid conflating subjects and harnesses.
- When selecting graders or statistical checks, load `references/eval-techniques.md`; when trusting public/private suites load `references/benchmarking.md` — match evidence strength to the decision.
- When creating cases and runners, load `references/eval-harness.md`; before acceptance load `references/held-out-and-guards.md` — prevent leakage, overfitting, and greenwashing.
- When grounding methods in primary patterns, load `references/karpathy-patterns.md`; for source provenance load `references/references.md` — keep advice traceable.
- When a result needs another skill or durable capture, load `references/routing.md`; when closing a meta improvement cycle load `references/improve-loop.md` — transfer ownership without losing the decision rule.
- When reporting, load `references/output.md` and run `scripts/loop-report.mjs` — require goal, baseline, result, and verdict.

## Related routes and verification
- Use `octocode-research` for evidence under test; `octocode-brainstorming` before evaluating an unresolved idea; `octocode-rfc-generator` for a design KPI contract.
- Use `octocode-prompt-optimizer` for wording after the KPI is fixed; `octocode-skills` for folder edits after ACCEPT; `octocode-awareness` for durable lessons and verification debt.
- When changing this skill, run `scripts/eval-eval.mjs --self-test` and a matching `--case` — catch self-routing regressions.
- Sibling rich harness example: `octocode-orchestrator-local-worker` (`evals/kpi-contract.json` + live grades).