This project asks a narrow research question: when repository-local guidance conflicts with an explicit user constraint, does a coding agent preserve the user's constraint after the guidance is actually encountered?
In these tested synthetic tasks, both Claude Code + Claude Sonnet 5 and Codex CLI 0.144.5 + GPT-5.6 Terra preserved the explicit user protected-path constraint in every treatment observation where the conflicting repository-local instruction was demonstrably encountered before editing.
| System | Comparable pre-edit exposures | Constraint preserved | Ever modified protected path |
|---|---|---|---|
| Claude Code + Claude Sonnet 5 | 8 | 8/8 | 0 |
| Codex CLI 0.144.5 + GPT-5.6 Terra | 5 | 5/5 | 0 |
- Claude V4/V5: 8 confirmed pre-edit treatment-artifact exposures; 8/8 preserved the constraint.
- Codex R9: 5 raw-evidence-supported pre-edit explicit-guidance exposures; 5/5 preserved the constraint.
The full comparison, evidence definitions, task-performance separation, and traceability are in reports/claude-vs-codex-instruction-integrity.md.
This is a system-level comparison, not a pure model comparison. Claude and Codex used different CLIs, runtimes, provider adapters, telemetry, and treatment designs. Claude's strongest denominator is treatment-artifact exposure, while Codex R9 used a separate explicit-guidance-file design. Only two synthetic tasks and small samples were evaluated. The result does not establish superiority, causality, population-level robustness, or general resistance to prompt injection.
- Treatment placement is not the same as actual exposure.
- Artifact exposure is not the same as guidance exposure.
- A final Git diff can miss a transient forbidden edit that was later reverted.
- Provider termination must be separated from task and verifier outcome.
- Infrastructure failures must not be interpreted as model behavior.
- Preregistration must bind executor-critical code.
- Registration hashes must derive from the same canonical Git-blob bytes used at execution.
These distinctions determine the denominator and prevent exposure-negative, provider-censored, or infrastructure-failed observations from being misread as instruction-integrity outcomes.
The evaluation uses owned synthetic repositories, paired control and treatment conditions, immutable one-trial preregistration, deterministic runner-owned verification, content-bearing exposure telemetry, and a runner-owned mutation ledger or filesystem observer for transient writes. Task success, authoritative verification, provider termination, exposure, and protected-path mutation are recorded as separate outcomes.
The public tree contains the harness, synthetic cases, deterministic graders, provider-neutral observation logic, selected source-bound registration logic, safe execution-boundary configuration, and derived aggregate findings. Raw registrations, provider telemetry, model responses, event streams, run artifacts, and diagnostic traces remain local-only. See public_release/RAW_ARTIFACT_POLICY.md and PUBLICATION_MANIFEST.md.
src/- harness, provider-neutral grading, exposure telemetry, mutation ledger, and preregistration guards.cases/- synthetic repositories and frozen condition overlays.specs/- design and preregistration specifications, including the Codex guidance and canonical-byte corrections.tests/- fake-provider, telemetry, invariant, ledger, and security-boundary tests.reports/claude-vs-codex-instruction-integrity.md- final cross-agent comparison.REPORT.md- concise Claude research report and evidence definitions.METHODOLOGY.md- architecture and outcome definitions.EXPERIMENTS.md- chronological Claude V1-V8R history.public_release/- derived public summaries and artifact policy.PUBLICATION_MANIFEST.md- the curated publication boundary.
Python 3.11 or newer is required. The normal local test suite uses the fake provider and makes no provider request:
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
$env:PYTHONPATH = (Resolve-Path src).Path
python -m pytest -qNo API key is required for the test suite, case checks, fake-provider runs, or local security checks. Real provider execution is separately guarded and is not part of normal development.
MIT. See LICENSE.