Acceptance Eval

# Acceptance Eval

> Pre-execution LLM-judgment: does this spec carry measurable, testable, complete acceptance criteria before work begins? Resolves the spec's acceptance section, critiques observability / testability / completeness (advisory `criteriaFindings`), flags user-visible behaviors with no covering test (advisory `coverageFindings`), and emits a confidence-rated `AcceptanceVerdict` (`MEASURABLE | NOT_MEASURABLE | INCONCLUSIVE`) with a rationale. Merge authority is derived in TypeScript, never trusted from the LLM: a high-confidence `NOT_MEASURABLE` blocks merge; every other verdict is advisory. The upstream twin of the `outcome-eval` ship gate — it keeps the downstream gate fed with judgable specs.

## When to Use

- On every spec entering the repo under `docs/changes/**` — triggered `on_pr` (and `manual`) — before execution begins.
- When you need a durable, structured answer to "can this spec's success be objectively judged later?"
- NOT for judging whether an implementation satisfied its spec post-execution (use `outcome-eval`, the downstream twin).
- NOT for authoring acceptance criteria — `acceptance-eval` reviews; humans own the thinking layer. It never writes the criteria it judges.
- NOT for rule-based floors (lint/architecture/entropy) or other craft ceilings — those run elsewhere.
- NOT when no judgable spec section exists — the verdict degrades to INCONCLUSIVE/advisory and never blocks.

## Process

### Phase 1: RESOLVE — Find the judgment section

The evaluator resolves the section internally via the fallback chain `## Success Criteria` -> `## User-Visible Behavior` -> `## Overview` (reusing `outcome-eval`'s resolver, not a fork), recording the match in `judgedAgainst`. No manual action — pass `specPath` and let `AcceptanceEvaluator` resolve. If no section is judgable, the verdict is INCONCLUSIVE/advisory.

### Phase 2: GATHER — Locate test evidence for coverage (optional)

For responsibility (b), supply test evidence so the judge can flag user-visible behaviors with no covering test: pass `testGlobs` (globs the tool reads) or `testContent` (snippets you already read). This evidence is optional: omitting it degrades `coverageFindings` to advisory-empty but NEVER affects the (c) measurability gate. Gather it when the spec describes user-visible behavior.

### Phase 3: JUDGE — Invoke the evaluator

1. Invoke the MCP tool `mcp__harness__acceptance_eval` with `{ specPath }` plus optional `{ testGlobs | testContent, model }`. The tool constructs `AcceptanceEvaluator` cli-side and returns the verdict; the supported v1 provider is the anthropic analysis provider (`ANTHROPIC_API_KEY`).
2. The LLM returns ONLY `measurability / confidence / criteriaFindings / coverageFindings / rationale`. `authority` is computed in TypeScript from `(measurability, confidence)` via `deriveAcceptanceAuthority` and is never read from the LLM — do not attempt to override it. The tool returns the verdict exactly as the evaluator derives it.
3. The call is degrade-safe: provider failure (incl. no `ANTHROPIC_API_KEY`), empty test evidence, or a missing judgable section yields INCONCLUSIVE/low/advisory. It never throws and never blocks.

### Phase 4: GATE — Render and (conditionally) halt

1. Render the verdict: `measurability`, `confidence`, `judgedAgainst`, `rationale`, `criteriaFindings` (a, advisory), and `coverageFindings` (b, advisory).
2. Authority rule (must match `deriveAcceptanceAuthority`): authority is `blocking` **iff** `measurability === 'NOT_MEASURABLE' && confidence === 'high'`; every other combination — including all `INCONCLUSIVE` and `MEASURABLE` cases, and all `medium`/`low` `NOT_MEASURABLE` — is `advisory`.
3. **On a blocking verdict: HALT before merge.** Report the missing measurable criteria and stop; the spec must not merge. Resolution requires a human adding measurable success criteria and re-running `acceptance-eval`.
4. On an advisory verdict: report the `criteriaFindings` and `coverageFindings` for human attention and proceed. Advisory findings do not stop the workflow.

## Harness Integration

- **`mcp__harness__acceptance_eval`** — MCP tool (the invocation surface). Inputs: `specPath` (required), `testGlobs` / `testContent` (optional (b) evidence), `model` (optional). The handler builds the cli `AnalysisProvider`, constructs `AcceptanceEvaluator`, and returns the `AcceptanceVerdict` with `authority` exactly as derived in TypeScript. (No graph persistence in v1 — the seam is documented in the tool header.)
- **Evaluator surface:** `AcceptanceEvaluator`, `deriveAcceptanceAuthority`, `acceptanceVerdictSchema`, `AcceptanceVerdict` are exported from `@harness-engineering/intelligence`. The section resolver is imported from `outcome-eval`, not duplicated.
- **Provider path (v1 supported):** the anthropic analysis provider (`ANTHROPIC_API_KEY`). When no provider is configured the call degrades to INCONCLUSIVE/advisory.
- **Relationship to `outcome-eval`:** `acceptance-eval` is the upstream twin. It guards spec measurability before execution; `outcome-eval` judges implementation satisfaction after. The same "authority is never read from the LLM" discipline spans both ends of the lifecycle.

## Known Limitations

- **Coverage findings (b) are heuristic and advisory.** Without `testGlobs`/`testContent` they are advisory-empty; even with evidence they are LLM-judgment over the behavior section plus located tests, not a graph-backed coverage map (deferred — see spec Decision D4). Heuristic misses are low-harm because (b) never blocks.
- **The (c) gate blocks only on high-confidence NOT_MEASURABLE.** A spec with weak-but-present criteria yields an advisory verdict, not a block — intentional, to avoid taste-blocks-merge false positives.
- **openai-compatible strict mode** is not the v1 path; the supported provider is anthropic/claude-cli.

## Success Criteria

See `docs/changes/harness-pm-persona/proposal.md` for the full criteria. This skill satisfies SC1 (skill exists, `tier: 2`, `type: rigid`, four-client artifacts), SC6 (blocking halt on high-confidence NOT_MEASURABLE; advisory-and-proceed otherwise), and SC7 (emits advisory `criteriaFindings` / `coverageFindings`).

## Rationalizations to Reject

These are common rationalizations that sound reasonable but lead to incorrect results. When you catch yourself thinking any of these, stop and follow the documented process instead.

| Rationalization                                                                                                   | Why It Is Wrong                                                                                                                                                                                                                                                        |
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "The model returned NOT_MEASURABLE and I agree, so I'll mark the verdict blocking and halt the spec."             | `authority` is never read from the LLM — it is `deriveAcceptanceAuthority(measurability, confidence)` computed in TypeScript. A `NOT_MEASURABLE` at `medium`/`low` confidence is advisory. Report the derived verdict exactly; do not upgrade it to blocking yourself. |
| "The success criteria are vague, so I'll rewrite them into measurable assertions and re-judge."                   | `acceptance-eval` reviews specs; it never authors the criteria it judges. Humans own the thinking layer. Report the criteriaFindings and halt if blocking — do not fix the spec to make the gate pass.                                                                 |
| "`ANTHROPIC_API_KEY` isn't configured, so I'll fail the gate to stay safe."                                       | Infrastructure noise must degrade to INCONCLUSIVE/advisory, never a block or a throw. A missing provider disables the gate; surface that to the human rather than manufacturing a halt.                                                                                |
| "There's no `## Success Criteria` section, but the Overview reads measurable enough, so I'll call it MEASURABLE." | Section resolution is the evaluator's fallback chain (`Success Criteria` → `User-Visible Behavior` → `Overview`) recorded in `judgedAgainst`. If nothing judgable resolves, the verdict is INCONCLUSIVE/advisory — do not fabricate a MEASURABLE verdict from prose.   |
| "The coverageFindings show a user-visible behavior with no test, so I'll block until a test exists."              | Coverage findings (b) are heuristic and advisory and NEVER affect the (c) measurability gate. Only a high-confidence NOT_MEASURABLE blocks. Surface the coverage gap for attention and proceed.                                                                        |

## Examples

### Example: NOT_MEASURABLE with high confidence (blocks)

**Input:** a spec whose only "success criteria" are "the feature works well" and "users are happy" — no observable, testable assertions.

**Verdict:**

```
measurability:   NOT_MEASURABLE
confidence:      high
judgedAgainst:   success-criteria
authority:       blocking
criteriaFindings:
  - { target: "'works well' / 'users are happy'", message: "No observable assertion — cannot be tested." }
rationale:       "The success section contains only subjective statements; nothing can be judged at outcome time."
```

**Action:** HALT before merge. Report the missing measurable criteria; the spec must not merge until a human adds them.

### Example: measurable criteria (advisory, proceeds)

**Input:** a spec whose Success Criteria list concrete, testable assertions; one user-visible behavior lacks an obvious covering test.

**Verdict:** `measurability: MEASURABLE confidence: high authority: advisory`, with one `coverageFindings` `{target, message}` object (the one user-visible behavior lacking a covering test) surfaced for review. The workflow proceeds.

## Gates

- **Authority is never read from the LLM.** The verdict's `authority` is always `deriveAcceptanceAuthority(measurability, confidence)` computed in TypeScript. If you find yourself letting the model assert blocking/advisory, STOP — that defeats the entire purpose of this gate.
- **Block only on high-confidence NOT_MEASURABLE.** `authority === 'blocking'` iff `measurability === 'NOT_MEASURABLE' && confidence === 'high'`. Every other combination is advisory. Do not halt on an advisory verdict.
- **Never author criteria.** `acceptance-eval` reviews specs; it does not write the acceptance criteria it judges. Humans own the thinking layer.
- **Never block on infrastructure noise.** A provider failure, an unparseable response, or a missing spec section must resolve to INCONCLUSIVE/advisory, never a thrown error or a block. The evaluator enforces this; do not reintroduce a hard failure in the wrapper.

## Escalation

- **Blocking verdict the author disputes:** the resolution is for a human to add measurable success criteria (or fix the section the resolver judged) and re-run `acceptance-eval` — not to override the gate.
- **Repeated INCONCLUSIVE on a real spec:** usually means no judgable section exists. Confirm the spec has a Success Criteria / User-Visible Behavior / Overview section.
- **No `ANTHROPIC_API_KEY` configured:** every verdict degrades to INCONCLUSIVE/advisory and nothing blocks. Surface this to the human — the gate is effectively disabled until a provider is configured.
- **Verdict seems wrong (false positive/negative):** capture the spec section and verdict and route to the maintainers; do not loosen the conservative-confidence prompt ad hoc.