Skip to content
Almathal Handbook

ADR-0012: Sanity Checks at Five Pipeline Stages

Accepted

Date: 2026-06-10

Context

A generation pipeline that runs straight through from input to output produces opaque failures: when something breaks at output, the cause could be anywhere upstream. Catching problems early saves tokens and provides clearer error messages.

LLM oversight is unreliable as a sole gate: LLMs reviewing LLM output can rubber-stamp their own work, and LLMs reviewing deterministic output can hallucinate problems that don’t exist.

Decision

Sanity checks run at five points in the pipeline, with determinism dominating and LLM oversight as a supplementary advisory signal:

  1. Compatibility Validation (deterministic, blocking) — pre-generation matrix walk
  2. Build Approval (user-driven, blocking) — human reviews LLM-generated summary
  3. Seam Validation (deterministic, blocking) — static analysis of Seam contracts
  4. Build Verification (deterministic, blocking) — compile, lint, test
  5. Output Review (LLM advisory, non-blocking) — AI Reviewer surfaces warnings

The first four are deterministic gates that block generation. The fifth is an advisory layer providing additional signal.

Rationale

  • Early detection of problems saves LLM tokens (Seam Validation alone may prevent doomed generations).
  • Deterministic gates produce repeatable, debuggable failures.
  • LLM oversight catches concerns mechanical checks miss (anti-patterns, security smells) but doesn’t gate on its judgment.
  • Defense in depth: a problem missed at one stage is likely caught at another.

Consequences

  • Each pipeline stage has well-defined inputs, outputs, and failure modes (documented in the Runtime Pipeline page).
  • The AI Reviewer uses a different LLM model than the Stitcher to reduce rubber-stamp risk.
  • Build Approval requires summary-generation logic in the Composer.
  • Verification harness (CI-style) must run automatically and quickly.

References

  • Runtime Pipeline
  • ADR-0009: Fully-Specified Seam Contracts (enables Seam Validation)
  • ADR-0010: Semver and Compatibility Matrix Versioning (enables Compatibility Validation)