fix(verify): both instrument false-positive mechanisms on the ORE corpus (#87) - #102
fix(verify): both instrument false-positive mechanisms on the ORE corpus (#87)#102micheldumontier wants to merge 1 commit into
Conversation
…pus (#87) Of the 366 ORE ontologies `verify-el` can check, 8 reported `Violated` and ALL 8 adjudicated to the instrument's own bugs rather than engine gaps — so the crate's headline claim ("a Violated is a real engine defect") was, corpus-wide, never once earned. Both mechanisms are fixed. F4 — `SubClassOf(owl:Thing, C)` never reached a Tseitin witness -------------------------------------------------------------- The saturator emits no fact for a nested existential body and gives its marker an EMPTY subsumer set BY DESIGN, so the witness element carried only itself and the checker reported the very `⊤ ⊑ C` axiom that should have closed it. On all four affected ontologies the violation count equalled the `⊤ ⊑ C` axiom count EXACTLY. The engine was right throughout: `rustdl subclass X C` answers `yes`, and the same reproducer with the nesting flattened away verifies clean — that pair is what isolates the marker path as the cause. `FiniteModel::intern` now closes a label containing NO named class under the ⊤-supers, collected from both `SubClassOf(⊤, …)` and `EquivalentClasses(⊤, …)` and closed through the REPORTED subsumer closure (not the axioms) so an engine that failed to derive a closure step still shows up as a violation. F5 — a truncated model reported `Violated` instead of `Unresolved` ------------------------------------------------------------------ A `BoundTripped` means `build_model` stopped early, so most axioms have no witness left and the violations are artifacts of the cut: 25369 / 20292 / 54304 / 92573 of them, on the order of the domain size itself. Now exit 3 (no verdict), not exit 2. BOTH FIXES ARE DELIBERATELY NARROW ---------------------------------- Closing EVERY label under the ⊤-supers, or downgrading on ANY build reason, would each read as the natural generalisation — and each would trade this crate's false-POSITIVE problem for a false-NEGATIVE one, which is strictly worse for an instrument whose purpose is finding defects. A named class whose closure lacks a ⊤-super IS a genuine D10 gap; a `LabelNotClosed` alongside a violation is still a real disagreement against whatever was checked. A GUARD TEST SURVIVED ITS OWN SABOTAGE -------------------------------------- The scoping canary was first written end-to-end, removing `C` from a named class's element via `test_only_remove_from_label`. That seam runs AFTER interning, so the check fails under either scoping and the test discriminated NOTHING — it passed the "close every label" sabotage it existed to catch. The rule is now pinned at unit level on `intern` directly, where it can be observed, and the end-to-end test was renamed to claim only what it actually checks. SABOTAGE: 5 run, 4 caught first pass, 1 survived and was closed. S1 close every label -> SURVIVED, then caught by the unit pin S2 drop the subsumer-closure step -> caught (1 test) S3 drop the EquivalentClasses arm -> caught (1 test) S4 revert the fix entirely -> caught (5 tests) S5 downgrade on any build reason -> caught (negative control) The F2 sentinel in `known_limitations.rs` is kept and INVERTED rather than deleted, as its own header instructs, and its "fixed, or merely untriggered by this shape?" question is answered by three further tests: a 3-deep nesting, the `EquivalentClasses` spelling, and a ⊤-super reachable only through the closure. F1 and F3 remain open and still reproduce. Verified: the 4 F4 ontologies now report `verified`, the 4 F5 ones `unresolved` with exit 3. Suite 1917 passed / 0 failed; clippy clean workspace-wide. Closes #87. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Closing as a duplicate — #87 was fixed in parallel and merged as 8bbadb7 / 5a9ea3a / 83bff7a while this was in flight. I compared the two implementations rather than assuming. The merged one is a superset of this one on every point but one:
The I verified the merged binary independently, on my own harness, on all eight ontologies #87 names — 4 Two things from this branch are worth keeping, and I am not re-landing either here:
Worth recording from this branch regardless: my first scoping guard survived its own sabotage — it removed a label entry via |
Closes #87.
Of the 366 ORE ontologies
verify-elcan check, 8 reportedViolatedand all 8adjudicated to the instrument's own bugs, not engine gaps — so the crate's headline claim
("a
Violatedis a real engine defect") was, corpus-wide, never once earned. Both mechanismsare fixed.
F4 —
SubClassOf(owl:Thing, C)never reached a Tseitin witnessThe saturator emits no fact for a nested existential body and gives its marker an empty
subsumer set by design, so the witness element carried only itself and the checker reported
the very
⊤ ⊑ Caxiom that should have closed it. On all four affected ontologies theviolation count equalled the
⊤ ⊑ Caxiom count exactly.The engine was right throughout, and the discriminating pair is what shows it:
rustdl subclass X Canswersyes, and the same reproducer with the nesting flattened away verifiesclean. It reproduces in ten lines:
FiniteModel::internnow closes a label containing no named class under the ⊤-supers,collected from both
SubClassOf(⊤, …)andEquivalentClasses(⊤, …)and closed through thereported subsumer closure rather than the axioms — so an engine that failed to derive a
closure step still shows up as a violation instead of being papered over.
F5 — a truncated model reported
Violatedinstead ofUnresolvedA
BoundTrippedmeansbuild_modelstopped early, so most axioms have no witness left andthe violations are artifacts of the cut: 25369 / 20292 / 54304 / 92573 of them, on the order
of the domain size itself. Now exit 3 (no verdict), not exit 2 (defect found).
Both fixes are deliberately narrow
Closing every label under the ⊤-supers, or downgrading on any build reason, each reads
as the natural generalisation — and each would trade this crate's false-positive problem
for a false-negative one, which is strictly worse for an instrument whose purpose is
finding defects. A named class whose closure lacks a ⊤-super is a genuine D10 gap; a
LabelNotClosedalongside a violation is still a real disagreement against whatever waschecked.
A guard test survived its own sabotage
Worth recording, because it nearly shipped. The scoping canary was first written end-to-end,
removing
Cfrom a named class's element viatest_only_remove_from_label. That seam runsafter interning, so the check fails under either scoping and the test discriminated
nothing — it passed the "close every label" sabotage it existed to catch. The rule is now
pinned at unit level on
interndirectly, where it can be observed, and the end-to-end testwas renamed to claim only what it actually checks.
Sabotage: 5 run, 4 caught first pass, 1 survived and was closed.
EquivalentClassesarmThe sentinel is inverted, not deleted
known_limitations.rspins F1/F2/F3 as current defective behaviour and its header instructsthat a newly-passing one be updated rather than removed, and be re-checked for whether the
gap is genuinely fixed or merely untriggered by that shape. Answered by measurement: three
further tests cover a 3-deep nesting, the
EquivalentClasses(owl:Thing, …)spelling, and a⊤-super reachable only through the closure. F1 and F3 remain open and still reproduce.
Verification
verified; the 4 F5 onesunresolvedwith exit 3CARGO_EXIT=0— captured directly, not through a pipe)cargo clippy --workspace --all-targets --all-features -- -D warningsexit 0verify-elsweep over all 1,920 ORE ontologies is running; I will post theverdict-distribution delta as a comment. The risk it covers is a new false
Violatedfrom the added labels — the fix only ever adds labels to marker-only elements, so that is
the direction to check.
🤖 Generated with Claude Code