Skip to content

adversarial-design.md

Latest commit

 

History

History
239 lines (167 loc) · 32.9 KB

adversarial-design.md

File metadata and controls

239 lines (167 loc) · 32.9 KB
Raw

Adversarial Factory Design

This document explains the roles, phases, and invariants that make up the ks factory's adversarial design, and the known limitations a user must keep in mind.

Why adversarial

The factory orchestrates LLM agents to implement software. An LLM left to its own devices is helpful by default - it will tell you the diff is good, the spec is clear, and the code is safe. Adversarial design assumes the opposite: that at every step something is wrong, and that the only way to find it is to commit a specific role to looking for it. Each role's prompt is framed to be skeptical, evidence-required, and gated by a check the LLM cannot lie its way around.

Calibration is the truth signal. The tests/test_calibration.py suite (Phase D of the hardening roadmap) feeds planted bugs to each role and measures detection rate. The whole adversarial design is only as good as that number says it is.

Roles

Role Module Prompt Phase What it catches
Architect / PRD red-team kstrl/decompose.py DECOMPOSE_PROMPT Spec Ambiguities, missing failure modes, unstated assumptions, undefined auth, ambiguous quantifiers. Closes every finding it raises as decided, assumed, spiked or escalated in a decisions register joined to spec_issues by id, and halts the pipeline via SpecBlockerError only on an escalation (#260).
Engineer kstrl/init_cmd.py (DEFAULT_PROMPT) + per-project scripts/kstrl/prompt.md (project-specific) Iteration Implements one story per iteration. Required to emit a ## Self-Critique block with >=3 substantive failure-mode bullets before declaring done (mechanically enforced by verify.check_self_critique when VerifyConfig.require_self_critique is True).
Mechanical verifier kstrl/verify.py (no LLM) Phase 1 PRD stories pass-marked, tests/typecheck/lint green, diff-scope and bad-pattern checks, optional dead-code / mutation / self-critique.
Code reviewer kstrl/review.py REVIEWER_PROMPT Phase 2 PRD criterion verdicts plus a separate concerns array (scope_creep, security_concern, test_quality, unrelated_change, dead_code, error_handling, copy_paste). Self-Critique block is stripped from the diff before review so the reviewer is not biased by the engineer's own failure-mode list.
Security reviewer kstrl/security.py SECURITY_PROMPT Phase 2.5 Threat-model framing: injection, auth_bypass, hardcoded_secret, unsafe_deserialization, broken_crypto, predictable_randomness, missing_input_validation, race_condition, SSRF, XSS, open_redirect, information_disclosure, denial_of_service. Each category mapped to OWASP Top 10 + CWE via SECURITY_CATEGORY_MAP.
Contract tester kstrl/contract.py (no LLM) Phase 3 Cross-component integration tests on merged tier branches. Failing tier identifies a breaker component, sent back through Phase 1+ for retry.
Knowledge distiller kstrl/knowledge.py DISTILL_PROMPT Pre-PR Captures durable facts about the artifact for downstream components. Runs after the Phase 2/2.5 gates pass and BEFORE the PR is created, so the distilled diff is the component's true delta (not polluted by the merge pulling main in). Voyager-style write gate: only fires when Phase 2 review passed. The placement is a named pipeline step (ComponentPipeline._phase_distill in kstrl/pipeline.py, decided in R7.3): moving it post-merge would break the true-delta invariant. tests/test_pipeline.py::TestDistillPlacement pins the pre-PR ordering.
Human checkpoint (E6) kstrl/factory.py (interactive) Pre-merge Optional. When FactoryConfig.pause_before_pr_merge=True and UI is interactive, prompts a human to approve or reject before PR creation. Off by default. Honoured only by per-component PR creation under ks factory (and ks serve via its factory invocations): ks run creates no PRs, and single_pr mode creates one aggregate PR without a checkpoint, so the gate does not apply in either case. Whenever the resolved config has the gate on while the checkpoint is unreachable - including when the L1/L2 autonomy bundle forces it on - run_factory emits a startup warning (#207).

Findings model (E3)

Every finding produced by Phase 2 (ReviewResult) or Phase 2.5 (SecurityResult) is converted into a typed Finding (kstrl/findings.py) before landing on Component.findings: list[Finding]. Consumers: pr.py renders findings into the PR body via render_findings_markdown (the legacy review_findings string is a fallback for legacy manifests); evolution.py::record_run serializes findings + a findings_summary aggregator into the journal.

The fields are:

Field Type Notes
phase str "review" or "security"
category str Reviewer concern category, security category, or "prd_criterion" for failed acceptance criteria
severity str Native to the role: "fail" / "advisory" (review), "critical" / "high" / "medium" / "low" (security)
location str file:line, file path, or "(entire file)"
explanation str Free text
suggestion str Optional
owasp, cwe str Populated for security findings via SECURITY_CATEGORY_MAP
tags tuple[str,...] Free-form; reserved for downstream consumers

Two consumers, two surfaces. Component.findings: list[Finding] is the typed surface, consumed by evolution.py::record_run for dashboards and trend analysis. Component.review_findings: str is the rendered surface, consumed by pr.py::build_pr_body for human-readable PR descriptions. They carry overlapping but non-identical information: the typed list has OWASP/CWE tags and structured filtering; the string has PASS-criteria confirmations, summary counts, and the criterion text as headers. Neither is a derived view of the other -- they are both populated from the same ReviewResult/SecurityResult and serve different downstream needs.

Infrastructure error semantics (E3-infra)

When a role's result has infrastructure_error=True (timeout, parse failure, agent crash), as_findings() emits a single synthetic Finding(phase=<role>, category="infrastructure_error", severity="critical") with is_infrastructure_error=True. This guarantees:

  • len(findings) == 0 always means "the role ran AND found nothing": a verifiably clean review.
  • [f for f in findings if not f.is_infrastructure_error] filters to the verified subset.
  • A consumer that checks only len(findings) > 0 to gate something will not accidentally pass an unverified component through.

Tag conventions (E3-tags)

Each Finding emitted by the factory path carries:

  • phase:<role> (review or security)
  • category:<X> (matching the category field)
  • For security: owasp:<bucket> and cwe:<id> when SECURITY_CATEGORY_MAP covers the category
  • For infrastructure errors: infrastructure

Tags let downstream consumers filter by taxonomy without re-parsing the field-level data.

Pipeline

spec.md
  -> [Architect] decompose + red-team + dispose
        -> manifest.json + per-component PRDs, then decisions.json
        -> SpecBlockerError if it ESCALATED (halt; owner answers)
  -> for each component (DAG order, optionally parallel):
       -> [Phase 0] feedforward (computational structural scan)
       -> [Engineer] iterate until COMPLETE
       -> [Phase 1] mechanical verify (incl. optional self-critique check)
       -> [Phase 2] code reviewer (criteria + concerns)
       -> [Phase 2.5] security reviewer
       -> [Knowledge distiller] pre-PR write
       -> [HITL checkpoint] if enabled
       -> create + merge PR
  -> [Phase 3] contract testing across merged tiers
  -> evolution journal recorded

Invariants

  1. Halt over heroics. Architect's SpecBlockerError stops the pipeline rather than proceeding with a vague spec. Mechanical verification failures retry up to FactoryConfig.max_retries then mark the component failed and cascade-skip dependents.
  2. Hard mode means hard fail. Phase 2 / Phase 2.5 in hard mode block on findings at or above the configured threshold. Infrastructure failures (agent crash, parse error) in hard mode count as failures, not silent passes (Phase A1 + E9).
  3. Latest-run-dir wins for facts. Knowledge files at .kstrl/knowledge/<component_id>/<run_id>/<fact_id>.md. A breaker retry naturally orphans the old run dir.
  4. No prompt injection through knowledge. Fact claims that match role markers (<system>, <|im_*|>), ignore previous instructions patterns, or ## Instructions headings are rejected at coercion time (Phase A1).
  5. No infinite cost. FactoryConfig.max_adversarial_calls is a hard cap across review + security + distill (Phase E4). Stream-size cap of 5MB per agent invocation prevents pathological output (Phase A5).
  6. Audit trail. Evolution journal records every component result. Concerns surface to EvolutionJournal.get_concern_hit_rate() for aggregate dashboards. Knowledge fact utilization is measured per component via knowledge.measure_fact_utilization() and RECORDED, to the fact_utilization_measured event and to component_result.knowledge_utilization in the journal, where EvolutionJournal.get_fact_utilization() aggregates it (#191). It is deliberately NOT on DistillResult, which covers the knowledge layer's write side: one measurement in one event, emitted on every path rather than only when distillation ran. It is measured against the prefix captured at engineer-submit time, not a rebuild: rebuilding after the distill pass counted facts this run had just written and the engineer never saw. Unmeasured is recorded as a distinct state from a measured zero, so a broken recorder cannot pass for "the engineer referenced nothing".
  7. Re-run, don't rebase (R7.5 merge-conflict doctrine). When a component's PR is CONFLICTING with base (GitHub's own verdict via gh pr view --json mergeable, surfaced as PrOutcome.merge_conflict), the factory does NOT rebase or merge-resolve the agent's output: a model resolving a conflict sees only textual hunks, not the sibling component's intent, and a "successful" rebase that silently breaks the sibling is worse than a loud retry. Instead the pipeline closes the conflicting PR with an audit comment, deletes its remote branch, clears the manifest's PR pointers, and re-runs the component through the fresh-base retry path (ComponentPipeline._retry_after_merge_conflict -> retry_or_fail(fresh_base=True)): the worktree AND branch are recreated from origin/<base>, which already contains the sibling changes that caused the conflict, so the engineer implements WITH the merged code in view. Conflict re-runs consume ordinary retries (max_retries bounds them) and record the pr:merge-conflict failure signature. Scope: the per-component PR flow only - single_pr mode defers PR creation to end-of-run (a conflict there is operator-resolved), and Phase 3 contract-tier merge conflicts are diagnostic (bisection blame), not integration merges.

Calibration

Calibration is the trustworthy verification path for "do the adversarial roles actually catch bugs." Without it, every "exhaustively_searched: true" claim is unverifiable.

To run:

KSTRL_RUN_CALIBRATION=1 KSTRL_CALIBRATION_MODEL=haiku uv run pytest tests/test_calibration.py -v

Each run executes every fixture KSTRL_CALIBRATION_RUNS times (default 3, R5.1) and writes tests/adversarial_fixtures/_results/baseline-<UTC>.json in the v2 format defined by kstrl/calibration.py: per-fixture consistency (fraction of completed runs that caught the planted issue; agent-infrastructure errors are excluded from the denominator, unparseable model output counts as a miss), per-role and per-category (per-CWE for security) detection rates, the run count, and the model id. Fixtures cover security (5), reviewer concerns (3), and vague specs (3), plus one non-halting allowedPaths fixture.

The fixtures themselves live in tests/adversarial_fixtures/{security,concerns,specs}/ with paired .meta.json files describing the planted bug and the must-detect category.

Threshold gates instead of hard asserts (R5.1)

A truth signal that is expected to be red is not a signal, so the suite no longer hard-asserts each single run. A fixture test passes when a majority of its completed runs detect the planted issue (FIXTURE_DETECTION_THRESHOLD = 0.5, i.e. 2 of 3 at the default run count): one flaky miss is reported as reduced consistency, a fixture that misses most runs fails the suite. Set KSTRL_CALIBRATION_RUNS=1 for a cheap single-run smoke (it degrades to the old hard-assert behavior and is too coarse for baseline capture).

Comparing baselines (H2's "compare" step, concretely)

uv run python -m kstrl.calibration compare \
  tests/adversarial_fixtures/_results/baseline-<old>.json \
  tests/adversarial_fixtures/_results/baseline-<new>.json

Exit code 0 = no regression, 1 = regression, 2 = usage/load error. Both v1 (pre-R5.1 single-run) and v2 files load. The codified thresholds live in one constants block at the top of kstrl/calibration.py with sizing rationale inline:

Constant Value Meaning
MAX_ROLE_DETECTION_DROP 0.15 A role's mean detection rate may not drop more than this between baselines. Sized so one run flipping on a 3-fixture role (drop ~0.11) is variance, an entire fixture going dark (~0.33) is a regression.
MAX_CATEGORY_DETECTION_DROP 0.40 Same per category (per-CWE categories usually hold one fixture: one run flip ~0.33 tolerated, two flips fail). Only meaningful at 3+ runs.
MIN_ROLE_DETECTION_RATE security 0.80, reviewer/architect 0.65, allowed_paths 0.50 Absolute floors on the new baseline so successive comparisons cannot ratchet a role downward.
FIXTURE_DETECTION_THRESHOLD 0.5 Majority-of-completed-runs gate used by the suite and by per-fixture detected.
DEFAULT_CALIBRATION_RUNS 3 Default runs per fixture for baseline capture.

Partial runs (a role present in the old baseline but not exercised in the new one) and cross-model comparisons produce warnings, not failures - the latter because a cross-model delta measures the model change, not a prompt change.

Kind synonyms (matcher de-brittling, R5.1)

The architect matcher's must_include_kind used to demand exact taxonomy labels, but every architect miss in the recorded 20260527 baselines was the planted issue reported under a sibling label (missing_detail instead of undefined_failure_mode / unstated_assumption) - a matcher artifact, not a detection failure. calibration.KIND_SYNONYM_GROUPS documents the one symmetric family that collapses: {missing_detail, unstated_assumption, undefined_failure_mode} ("the spec is silent about X"). ambiguity, contradiction, out_of_scope_creep, and other still require exact labels - ambiguity is about vague language that IS present, not absence. Exact-label matching is still recorded in run details (exact_kind_match=) as a non-gating signal so taxonomy drift stays visible.

Reviewer-family override (R7.1)

To measure the same-family vs cross-family correlated-miss delta, the calibration runner accepts a reviewer-family override that applies to the reviewer and security roles only (the architect keeps the base calibration agent - rotation applies to reviewers, not the spec red-team):

# Baseline 1: same family end to end
KSTRL_RUN_CALIBRATION=1 KSTRL_CALIBRATION_MODEL=haiku \
  uv run pytest tests/test_calibration.py -v

# Baseline 2: reviewer + security roles on the second family (codex CLI).
# Two codex-specific requirements (measured; see the gotcha note below):
#   - pin KSTRL_CALIBRATION_REVIEWER_MODEL (haiku is claude-only -> HTTP 400)
#   - --basetemp INSIDE a git repo (codex refuses a non-git cwd)
CODEX_REPO=$(mktemp -d); git init -q "$CODEX_REPO"
KSTRL_RUN_CALIBRATION=1 KSTRL_CALIBRATION_MODEL=haiku \
  KSTRL_CALIBRATION_REVIEWER_AGENT_TYPE=codex \
  KSTRL_CALIBRATION_REVIEWER_MODEL=gpt-5.5 \
  uv run pytest tests/test_calibration.py -v --basetemp="$CODEX_REPO/pt"

# Compare (the cross-model warning is expected: the delta measures the family change)
uv run python -m kstrl.calibration compare \
  tests/adversarial_fixtures/_results/baseline-<same-family>.json \
  tests/adversarial_fixtures/_results/baseline-<cross-family>.json

KSTRL_CALIBRATION_REVIEWER_MODEL optionally pins the reviewer model within the overridden family. Override runs record their model id as <base>+reviewer:<type>/<model> so compare surfaces the family change as its cross-model warning instead of hiding it.

Gotcha (codex 0.134 on a ChatGPT account, measured 2026-07-20): the cross-family command above needs two things the same-family run does not, or it silently records a miss on every fixture. (1) Pin KSTRL_CALIBRATION_REVIEWER_MODEL to a valid codex model (e.g. gpt-5.5). Without it the override forwards KSTRL_CALIBRATION_MODEL (haiku) to codex exec -m haiku, which a ChatGPT-account codex rejects with HTTP 400 - but codex still exits 0 and prints the error as output, so the parser sees no findings and scores a clean miss (not an infrastructure error). (2) Add --basetemp=<dir inside a git repo> to pytest (an empty throwaway git init dir works; pytest empties the basetemp, so make it a SUBDIR of the repo, not the repo root). Calibration runs each agent in an empty tmp_path, and codex exec refuses a non-git cwd (Not inside a trusted directory and --skip-git-repo-check was not specified) and fast-exits - again a silent miss. The claude CLI has neither restriction, so the same-family run needs no special handling; the runbook's plain 2c command only works for a claude-family reviewer.

Recorded baselines

  • Same-family (2026-07-20, haiku): tests/adversarial_fixtures/_results/baseline-20260720-113835.json. Reviewer 4/4, security 6/6, security_hard 4/4 - all detected 3/3, reviewer/security false-positive rate 0.0 (0/8 clean fixtures). This is Baseline 1 above.
  • Cross-family (2026-07-20, codex/gpt-5.5 on reviewer+security, haiku architect): captured; artifact baseline-20260720-123959.json (label haiku+reviewer:codex/gpt-5.5). NOT committed to the repo results dir: it is not the canonical baseline and would trip test_repo_baselines_match_default_model, which asserts the newest committed baseline uses the default model. compare same-family -> cross-family = PASS (cross-model warning expected). Deltas, 3 runs each, 0 infrastructure errors:
    • reviewer 1.00 -> 1.00; security_hard 1.00 -> 1.00; architect unchanged (stays haiku in both).
    • security detection 1.00 -> 0.94: codex missed sec-05-broken-jwt-verify (auth_bypass) in 1 of 3 runs (2/3); every other security fixture stayed 3/3.
    • security false-positive rate 0.0 -> 0.25: codex false-flagged sec-neg-02-constant-time-compare in 2 of 3 runs (still under FP_RATE_MAX 0.34); reviewer FP stayed 0.0.
    • Interpretation: no measurable correlated-miss BENEFIT is visible on this fixture set, because both families already sit at the detection ceiling (same-family catches 100%, including all 4 "hard" positives), so there is nothing for the second family to additionally catch. The only cross-family deltas are codex being marginally noisier (one flaky miss, one extra false positive). A real correlated-miss delta requires fixtures that a single family sometimes MISSES - the same gap as the R5.2 hard-positive finding (4/4 caught = not actually hard). So the rotation's effect size remains unevidenced until the fixtures have genuinely-missable cases, even though both baselines are now recorded.

Hard-positive hardness: measured, and genuinely caught (R5.2, 2026-07-20)

The R5.2 acceptance check wants security_hard.detection_rate < 1.0, on the premise that a baseline catching all four "hard" positives means they are too easy. The 2026-07-20 capture returned 1.0 (all four caught 3/3), so this was investigated before concluding the fixtures are defective:

  • The matcher is not lenient. security_caught requires a finding whose category is in the fixture's category_any_of, AND whose location contains evidence_path_contains (the specific vulnerable file), AND whose severity meets the floor. A finding elsewhere, or of the wrong category, does not count.
  • The catches are genuine. Re-running each hard fixture and reading the model's explanation shows mechanism-level reasoning, not category guessing: multihop-authz ("deletes a comment by id without verifying it belongs to the authorized post"), second-order-injection ("validated at entry but quotes allowed, then interpolated at the analytics sink"), TOCTOU ("non-atomic read-check-write, not in one transaction, under concurrent requests"), timing-oracle ("returns on first mismatch; recover the signature byte-by-byte by latency").
  • Even tell-free variants are caught. Rewriting the timing oracle to remove its tell - replacing the hand-rolled _secure_equals early-return loop with a plain return expected == provided (still non-constant-time, but innocent-looking) - haiku still flagged the timing side channel 5/5, naming constant-time comparison each time.

Conclusion: these are well-designed subtle bugs; haiku is simply a competent reviewer for OWASP-classic categories, and detection_rate < 1.0 is not reachable for them without contriving scenarios that hide the security-relevant operation itself. The hard positives keep their value as a regression net (the MIN_ROLE_DETECTION_RATE / MAX_ROLE_DETECTION_DROP floors still fire if a prompt edit degrades them) and as a cross-family / weaker-model gradient; they are recorded but NOT gated. On this evidence R5.2's original detection_rate < 1.0 bar was dropped and R5.2 was closed (2026-07-20, user decision): the hard positives stay as genuinely-subtle, measured-not-gated fixtures protected by the detection-drop floors. See remediation-roadmap R5.2.

Model drift (R5.5, H2-extended)

Baselines record the model id, and an always-run structural test (tests/test_calibration.py::TestFixtureStructure::test_warns_when_calibration_model_differs_from_newest_baseline) warns - never fails - when KSTRL_CALIBRATION_MODEL differs from the newest baseline's recorded model. H2 extended: calibration re-runs on model change, not just prompt change; a detection rate measured against an older model does not transfer.

Feedforward vs knowledge (E7)

Two memory surfaces exist for the implementing agent. They look similar but serve different jobs:

  • Feedforward (kstrl/feedforward.py) is computed fresh each iteration. Walks the worktree, builds a module map with LOC counts, lists public interfaces from __init__.py / __all__, infers a dependency graph from imports, and extracts conventions from pyproject.toml / package.json / etc. No LLM, no persistence. Used to ground the implementing agent in the current code shape.
  • Knowledge (kstrl/knowledge.py) is distilled by an LLM after a component completes and persists across runs. Stored at .kstrl/knowledge/<component>/<run>/<fact>.md. Three-tier retrieval (core / dependency / sibling) injects relevant facts into the prompt of downstream components.

The overlap: both can describe what a component exports. The distinction:

  • Feedforward describes what exists at this instant (computationally extracted).
  • Knowledge describes what was learned about an artifact's contract or invariants (LLM-distilled, durable).

If a feedforward entry says auth.middleware.verify_token(token: str) -> User, that's the current signature. If a knowledge fact says "the middleware rejects expired tokens at the handler layer, before the route guard runs," that's the behavior the LLM extracted from passing tests + the diff. They complement, but neither replaces the other.

Dependency scope (E8)

The knowledge layer's "Dependencies" tier defaults to direct scope: only facts from Component.dependencies (the import surface declared in the manifest) appear in the full-text tier. Transitive dependencies still surface in the sibling summary tier (first-sentence only).

Rationale: the typical reason a component needs full-text facts about a transitive dependency is that the manifest is missing a direct edge - i.e. the architect under-specified imports. Forcing the user to add the edge is better than silently injecting every transitive ancestor's facts into every downstream prompt. For projects that genuinely need the old behavior, KnowledgeConfig.dependency_scope = "transitive" (or KSTRL_KNOWLEDGE_DEPENDENCY_SCOPE=transitive) restores it.

Telemetry for the direct-vs-transitive gap (E8-telemetry)

Switching to direct scope can silently drop facts that downstream components were relying on. To make that visible, build_knowledge_context writes a per-component event to <knowledge_root>/_e8_dependency_scope.jsonl every time it excludes one or more transitive deps. The event records excluded_dep_count and withheld_fact_count. Read via read_dependency_scope_telemetry(knowledge_root) -> list[dict].

Healthy state: empty file. Persistent non-zero values per build are the signal that direct scope is dropping information real workflows need, and the architect should be asked to make the missing edges explicit (or dependency_scope=transitive re-enabled).

Known limitations

  1. Correlated failure (partially mitigated by R7.1 rotation). The review and security phases now default to the OPPOSITE model family from the engineer when that CLI is available (user decision 2: the OpenAI family via the codex CLI reviews Claude-engineered code; a codex engineer flips the default to claude-code). Explicit reviewer config always wins; when the cross family's CLI is missing (or the engineer runs a custom command whose family is unknowable) the factory prints a homogeneity warning naming the self-preference risk and falls back to the old same-family behavior. Every reviewer-produced Finding carries a model:<id> tag, the PR body's findings sections name the reviewer model, and the journal serializes the tag - so same-family and cross-family review outcomes stay attributable and measurable. What REMAINS correlated: the architect, engineer, and knowledge distiller still run on the primary family, so a spec misreading or implementation blind spot shared by that family is not caught by rotation - treat architect+engineer+distiller agreement as one data point. The correlated-miss delta is measured, not assumed: capture same-family vs cross-family calibration baselines (see "Reviewer-family override" below) before trusting the rotation's effect size.
  2. exhaustively_searched is self-reported. Both reviewer and security results expose the flag, but it cannot be verified at runtime. The trustworthy signal is calibration rate, not the flag.
  3. Fact-utilization is a lower bound. measure_fact_utilization uses a 30-character case-insensitive substring match, so an LLM that paraphrases a fact it genuinely used scores as not referencing it. Three defects that broke that lower-bound property are corrected rather than merely documented (#191 and its review): only ADDED diff lines and the progress log are searched, so deleting the code that expressed a fact - or editing near it - no longer scores as using it, which was a false POSITIVE and the one direction a lower bound must not have; the measurement runs as soon as a diff is fetchable rather than in the distill phase, so components failing verification, review, or security are in the sample instead of only those passing every gate; and the counts are split per prefix tier, so sibling first-sentence summaries no longer silently inflate the denominator - read core_referenced / core_injected for the ratio about the component actually being built. measured: false is reserved for a real inability to measure and always carries a reason. The lexical match itself remains the standing caveat.
  4. Calibration baseline is non-deterministic. LLMs vary; the suite now runs each fixture KSTRL_CALIBRATION_RUNS times (default 3) and reports per-fixture consistency (R5.1), but 3 runs is still a small sample - treat a consistency of 2/3 as "flaky", not as a precise 0.67.
  5. Windows is not supported for concurrent worktrees. fcntl.flock is POSIX-only (Phase A4); on Windows the lock is silently skipped and concurrent factory invocations against the same worktree directory can clobber each other.
  6. The fact-injection prompt is trusted code. A future model that ignores the engineer prompt's "treat as ground truth" framing could be misled by injected facts. The Phase A1 sanitizer is a defense-in-depth pattern, not a guarantee.

Process: how this design stays trustworthy

H1 of the hardening roadmap: the assistant does not run /code-review on its own code. The user, or /code-review ultra, is the gating reviewer for changes that touch this design.

H2: when an adversarial prompt changes, calibration is re-run. A prompt edit without a calibration delta is treated as untested.

H3: every adversarial prompt has a *_PROMPT_VERSION semver constant next to its body and a (hash, version) snapshot in tests/test_prompt_versions.py::_EXPECTED_SNAPSHOTS. The enrolled set is _PROMPTS in that file, nine as of #260 (DECOMPOSE_PROMPT, REVIEWER_PROMPT, SECURITY_PROMPT, DISTILL_PROMPT, VERIFY_COMMANDS_PROMPT, REPO_CHANGE_SOURCE_PROMPT, PASTED_CHANGE_SOURCE_PROMPT, DEFAULT_PROMPT for the engineer role, and DECISIONS_CONTEXT_PROMPT). It is not only the LLM-driven role prompts: the engineer template (init_cmd.DEFAULT_PROMPT, scaffolded into per-project scripts/kstrl/prompt.md), the verification-commands block, and the two reviewer change-acquisition bodies are all enrolled on the same terms.

The joint snapshot plus its companion tests catch six drift modes:

  1. Prompt edit without snapshot bump: hash differs from recorded hash, test fails.
  2. Version constant change without snapshot bump: live version differs from snapshot version, test fails.
  3. New *_PROMPT added without enrollment: test_no_unenrolled_prompt_constants AST-walks kstrl/ and fails on any unprotected prompt. The walk keys on the target NAME, at any nesting depth, plus a value it cannot prove is a non-string. Since #299 the value test is default-deny: a *_PROMPT-suffixed name is flagged unless its value is a non-string literal, a collection display, or a call to a builtin that cannot return a string. So dedent(...), .strip(), %, SEP.join(...), A + B, a ternary and a bare alias are all caught, and frozenset({...}) is not. It remains blind to instruction text never bound to a *_PROMPT name at all; that residual is recorded in the H3-NOTE of tests/test_prompt_versions.py.
  4. Enrolled but never hash-checked: the per-prompt check is parametrized over _PROMPTS, so enrollment alone puts a prompt under snapshot. VERIFY_COMMANDS_PROMPT sat enrolled and unchecked from #261 to #299 because that check used to be hand-written per prompt.
  5. Enrolled but not the text that ships: test_renderer_renders_the_enrolled_body patches each constant and asserts its production renderer returns that and nothing else, so a constant cannot rot into an orphan, get wrapped in unhashed words, or be truncated. It covers the eight prompts that have a renderer; DEFAULT_PROMPT is in _RENDER_EXEMPT because ks init writes it to disk verbatim and nothing interpolates it, so it has no render step to orphan, and its reach is covered by H3b's scaffold ledger instead. test_every_prompt_has_a_renderer stops the table falling behind the enrolled set. test_the_real_enrolled_body_renders additionally renders each real body unpatched, so a template whose placeholders the renderer does not supply fails here rather than at run time.
  6. Enrolled body dropped by its caller: test_change_source_reaches_the_role drives run_review and run_security_review and asserts the change-acquisition body still reaches the prompt each role is sent, which a leaf render guard cannot see. This covers REPO_CHANGE_SOURCE_PROMPT only. PASTED_CHANGE_SOURCE_PROMPT gets the leaf guard alone, and deliberately: its only callers are the calibration fixtures and this suite, so there is no production caller to drop it.

What H3 does NOT catch: instruction text that is never bound to a *_PROMPT name at all, returned straight out of a function or bound to a local called something else. #299 hoisted the two known instances by hand; docs/adversarial-roadmap.md H3 states the hoisting rule, and keeping to it is reviewer discipline rather than a check. tests/test_prompt_versions.py H3-NOTE records this residual, and issue #303 tracks the sites the H3a sweep found.

The audit trail is the PR diff with prompt body + version constant + snapshot tuple all moving together. That is what makes the H2 calibration step a real gate rather than a polite suggestion. H3 cannot prevent a determined developer from leaving the version pinned while updating both hash and snapshot to the previous version number; that bypass requires explicit deception in the snapshot file and is the irreducible limit of code-side enforcement.

H4: when reporting "tested" or "verified", be explicit about what was checked vs. what was assumed. Smoke tests are presence checks; calibration is behavior verification.