Skip to content

Replace SOMA with the LexCAT lexical runtime (v0.0.14) - #1

Draft
darthtrevino wants to merge 6 commits into
githubnext:mainfrom
darthtrevino:migrate-soma-to-lexcat
Draft

Replace SOMA with the LexCAT lexical runtime (v0.0.14)#1
darthtrevino wants to merge 6 commits into
githubnext:mainfrom
darthtrevino:migrate-soma-to-lexcat

Conversation

@darthtrevino

@darthtrevino darthtrevino commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown

Replaces the SOMA lexical runtime with LexCAT (msr-central/LexCAT) v0.0.14. Clean cut — SOMA is
removed entirely rather than kept behind a flag, and platforms without a LexCAT binary now fail with
an actionable error pointing at WIKIKB_LEXCAT_BIN.

Why

LexCAT is SOMA's information-retrieval engine, extracted into its own repository. SOMA itself remains
maintained — this is a re-targeting onto the engine's new home, not a move away from an abandoned
dependency.

WikiKB only ever used SOMA for lexical retrieval, so depending on the extracted engine directly drops
the surface area WikiKB never touched and puts it on the codebase where retrieval work now happens.
That also makes the standalone engine's CLI contract WikiKB's contract, which is where the porting
effort actually went: eleven DX gaps surfaced during the port were filed upstream, and all eleven
shipped across v0.0.12 – v0.0.14 before this branch landed.

What changed

Retrieval integration (tools/wikikb-local/src/main.ts)

  • Query goes through lexcat query --json, so hits carry text, doc_id, chunk_id, score and
    a payload block as structured output rather than prose. SOMA already returned chunk text in its
    JSON payload; early LexCAT did not, which is what upstream #216 added, so this is parity restored
    rather than a new capability.
  • Indexing goes through build --json / sync --json as of v0.0.14 (#237), so the chunk and term
    counts are read as fields instead of regex-matched out of a prose summary line.
  • The one field lost in the swap is SearchHit.community. SOMA grouped hits under
    communities/topics and WikiKB stamped the community id onto each hit — but nothing ever read it
    back, so dropping it changes no behaviour. LexCAT is a purely lexical engine and has no equivalent
    grouping; if WikiKB ever wants that, it would be new work, not a regression to restore.
  • Wiki metadata (path, title, revision) moves into per-document YAML frontmatter and comes back on
    every hit as payload.fields. build --frontmatter payload attaches it to all chunks of a split
    document, so attribution survives chunking.
  • Incremental reconcile uses lexcat sync, which re-uses the config baked into the index. Full
    rebuilds are reserved for config/schema changes and --force.
  • assertIndexIsQueryable guards every build and sync and now checks the term count as well as the
    chunk count. Neither an empty corpus nor a vocabulary the analyzer collapsed to nothing is an error
    to LexCAT — both write a valid index and answer every later query with hits: [] at exit 0 — so an
    index that cannot be searched is rejected where it is built rather than silently at query time. A
    chunk count alone would not catch the second case: an index can hold every chunk and still be
    unsearchable.

Vendoring (vendor/lexcat/)

Five platforms, up from four: linux-arm64, linux-x86_64, macos-arm64, macos-x86_64,
windows-x86_64. Every archive is verified against the upstream SHA256SUMS and pinned by digest in
manifest.json. win32/arm64 has no native binary upstream and falls back to x64 emulation via
selectLexcatArtifact.

vendor/soma/ is deleted. The LICENSE carve-out is retargeted from SOMA to LexCAT.

Runtime cache publishing (extractLexcatArtifact / publishLexcatRuntime)

CI caught a race here that is worth calling out, because the code is inherited rather than new.

Vendored archives are extracted into the per-user cache on first use. The published runtime used to
be installed with an rmSync of the target directory followed by a rename into its place. Extraction
was already safe — each process gets its own .extract-* scratch directory — but the publish step
was not: when two wkb processes extract concurrently, the loser of the race deletes the winner's
already-verified install directory before renaming its own copy over it. A sibling holding a path
into that directory then spawns a binary that no longer exists:

LexCAT failed to start: spawnSync /tmp/.../runtime/lexcat/v0.0.13-linux-x64/lexcat ENOENT

Publishing is now a bare rename. Renaming onto a populated directory fails with ENOTEMPTY on POSIX
and EEXIST/EPERM on Windows, which is exactly the signal that another process got there first; if
the runtime already in place matches the pinned digest it is adopted as-is rather than replaced. A
directory that is present but does not match is swapped aside and retried, so a corrupt cache still
self-heals.

This pattern came from extractSomaArtifact, which has the identical rmSync/renameSync sequence
on main today. It stayed latent there because SOMA's model and preset setup spread the two
processes far enough apart to hide the window; LexCAT starts fast enough to actually hit it. The
same race therefore exists on main and is not fixed by this PR — worth a separate fix if the SOMA
integration is staying around.

The regression test that caught it raced only two processes and reproduced the failure about a
quarter of the time, which is why three consecutive clean local suite runs missed it. It now races
four, which reproduces reliably, and the scratch-directory assertion covers the .stale-* path the
retry introduces.

Everything else

CI (lexcat-runtimes job), the release packager and validator, install.sh, the agentic workflow
templates (both .md sources and generated .lock.yml), and docs. docs/configuration.md gains a
"Runtime And Cache" section covering the frontmatter/--json contract, sync semantics, and the
chunk/term guard.

The one workaround this PR carried is now gone

Earlier revisions of this branch generated a lexcat.toml pinning analyzer_min_vocab = 1. LexCAT's
vocabulary floor counted documents where the reference implementation counted occurrences
(upstream #229 /
#233), so once chunk count greatly exceeded
document count every rare term — the highest-value lookup in a knowledge base — was pruned and
returned zero hits at exit 0.

v0.0.14 defaults the floor to 1, so the generated config file is deleted. Measured on an identical
66-chunk corpus with no config file at all:

runtime build report rare-term query
v0.0.13 (default) 8 terms 0 hits, exit 0
v0.0.14 (default) 22 terms 1 hit

The end-to-end suite pins this through the whole stack: a corpus containing the token zarfblitz
exactly once is retrieved by name after a build, after an incremental sync, and after a forced
rebuild — with no configuration file anywhere.

Dropping the config file cannot silently regress an operator-supplied WIKIKB_LEXCAT_BIN that is
older than v0.0.14, because --json on build/sync is itself a v0.0.14 flag: v0.0.13 rejects it
with error: unexpected argument '--json' found and exit 2, so the run fails loudly instead of
quietly losing recall. Verified against the shipped v0.0.13 binary.

Verification

check result
npm run check:offline (Linux) 44/44, 0 skips
End-to-end CLI drive on the real vendored binary (Linux) 29/29
Vendored windows-x86_64 binary, native Windows digest, --version, build --json, rare-term query, frontmatter payload all verified
npm run bundle:check clean, 112 entries, reproducible
node tools/validate-release.js 0 errors, 87 files scanned
Vendored blobs vs upstream SHA256SUMS 5/5 match
v0.0.13 → v0.0.14 upgrade path stale index detected, rebuilt, restamped, queryable
Runtime-publish race, 4 racers (Linux) 2/8 failures before the fix, 0/12 after
GitHub Actions, all 7 jobs green on c9f1f65

The e2e drive runs the real vendored binary against a local fixture — no network, no mocked CLI. It
covers registration, build, retrieval, rare-term recall, incremental reconcile, forced rebuild,
status/tags, config round-trip, the empty-corpus and missing-runtime guards, and zero-hit handling.

Index schema is unchanged at 11. INDEX_CONFIG_VERSION is deliberately not bumped: the old
explicit min_vocab = 1 and the new upstream default are semantically identical, and
runtimeCompatibility() already keys the cache on the runtime version, so every existing index is
invalidated and rebuilt on upgrade regardless. That path is exercised in the table above rather than
assumed.

One caveat on reading that table: the runtime-publish race was intermittent, so the earlier clean
44/44 runs were not evidence of its absence. The before/after numbers come from running the
affected test in a loop rather than once.

Two CI regressions this branch introduced

The first three pushes of this branch failed CI. Both causes were mine, not pre-existing on main
— I checked by running the same linters against origin/main, which is clean.

  • Workflows and shell scripts. actionlint runs shellcheck over every run: block. The JS
    template literals inside the single-quoted node -e programs read as shell parameter expansions,
    tripping SC2016. Shellcheck cannot distinguish the two, so the fix is string concatenation.
  • LexCAT Windows x64. tools/wikikb-local/wkb is an extensionless bash script with no Windows
    shim, so Windows cannot spawn it and spawnSync returns status: null. I had gated the two
    vendored tests on artifact presence alone, and a win32/x64 artifact exists, so they ran and
    failed. main gates the equivalent SOMA tests on platform for exactly this reason; this restores
    that. Windows keeps real coverage — the job drives the vendored binary through the LexCAT CLI
    directly (digest, --version, build/sync --json, query).

Both fixes were reproduced locally first: pinned actionlint 1.7.12 + shellcheck 0.10.0 flagged the
exact two lines CI reported and now pass, and the Windows job's exact node --test command was run
natively on Windows to confirm it exits 0.
The live integration suite (test:wkb:integration) was not run; it needs WIKIKB_TEST_REPO plus a
token and writes to a real wiki.

Upstream issues from this port

All eleven are closed and shipped:
#216 (query --json),
#217/#229/#233
(vocabulary floor),
#218 (frontmatter passthrough),
#219 (platforms + SHA256SUMS),
#220 (sync),
#221/#232 (--version),
#237 (machine-readable build/sync output),
#238 (sync omits the term count).

Review notes

  • .gitattributes line vendor/lexcat/** -text is load-bearing — without it core.autocrlf
    corrupts the archives and the notices digest.
  • Editing vendor/lexcat/THIRD_PARTY_NOTICES.txt requires recomputing notices_sha256 in the
    manifest.
  • Draft because it needs a maintainer to confirm the LexCAT redistribution authorization is recorded
    the same way the SOMA carve-out was, and that it covers the fork this branch is pushed from.

darthtrevino and others added 6 commits August 25, 2026 11:02
@darthtrevino
feat(retrieval): replace SOMA with the LexCAT lexical runtime
57fe5ee 
WikiKB now indexes and retrieves through LexCAT 0.0.11, a model-free BM25
engine, instead of the model-based SOMA runtime. This is a clean cut: SOMA
and its pinned embedding model are gone, along with WIKIKB_SOMA_MODEL_DIR
and the model install/lock machinery.

LexCAT has a much narrower contract, so the integration changed shape:

- `lexcat query` prints only `[score] chunk_id`, so chunk text is read back
  from the index SQLite file via node:sqlite, and titles, wiki paths, and
  dates are rejoined from a corpus manifest kept beside the corpus.
- LexCAT's default `analyzer_min_vocab = 2` silently yields zero hits on
  small corpora, so WikiKB generates a config pinning it to 1 and asserts
  the built index is non-empty rather than failing at query time.
- LexCAT has no frontmatter handling and indexes every byte as body text,
  so staged documents now carry prose only.
- There is no incremental build, so each build writes to a scratch file and
  is renamed into place, leaving the previous index intact when it fails.

Vendored archives drop from ~50 MB to 11.4 MB but cover three platforms
instead of five; linux-arm64 and windows-arm64 now fail with an error
pointing at WIKIKB_LEXCAT_BIN.

node:sqlite raises the floor to Node 22.5.0, which is enforced across
package metadata, workflow templates, the installer, and the validator.

Vendor archives are marked binary so autocrlf cannot invalidate the pinned
notice checksum, are repacked with the executable bit set, and are now
reproducible.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@darthtrevino
Adopt LexCAT v0.0.12
e8e9861 
Re-vendors all five upstream 0.0.12 executables (adds linux/arm64 and
darwin/x64) and rebuilds the retrieval integration on the contract they
add, replacing the workarounds v0.0.11 forced.

Retrieval now reads `query --json`, so chunk text, score, and doc id
come back with the ranking instead of being rejoined by binding against
LexCAT's private SQLite schema. That removes the node:sqlite dependency
and with it the Node >=22.5.0 floor, which returns to >=22.

Wiki identity is staged as YAML frontmatter on each corpus document.
LexCAT strips it from the indexed text and returns it on every chunk of
a split document, so the side corpus manifest is gone.

Reindexing an index built by the current contract now uses `lexcat
sync`, which reconciles only added, changed, and removed documents.
`index --force` still rebuilds from scratch. Both paths keep writing
to a scratch index that is swapped in only after it verifies.

WikiKB keeps generating lexcat.toml with analyzer_min_vocab = 1. The
0.0.12 corpus cap only avoids the total wipeout on a tiny corpus: a
corpus of a few long documents still loses nearly its whole vocabulary
at the default, and retrieval then returns no hits and exits 0.
Reproduced on this binary at 23 chunks from 2 documents: 9 terms
indexed and a rare term unfindable, against 408 terms with the floor
at 1. Reported upstream, along with --version reporting 0.0.0 on every
platform, which is why the manifest checksum stays the version pin.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@darthtrevino
Adopt LexCAT v0.0.13
3a6fca0 
Re-vendors the five v0.0.13 runtime archives and pins them by checksum.
All five executables round-trip byte-identical to the upstream SHA256SUMS.

v0.0.13 stamps the real release version into the published binaries
(upstream #234), so `--version` is now a trustworthy pin. The release test
and the CI smoke assert the manifest semver alongside the index schema
instead of the schema alone.

The index schema is unchanged at 11 and WikiKB's generated build config is
unchanged, so warm indexes survive the upgrade: a 0.0.12-built index was
verified to query, reconcile incrementally under `sync`, and retain its
payload fields under 0.0.13. INDEX_CONFIG_VERSION is therefore not bumped.

The analyzer vocabulary floor is still not fixed upstream (#233, root cause
#229), reconfirmed against the shipped 0.0.13 binary: a 2-document corpus
that splits into 67 chunks indexes 8 terms and returns no hits at exit 0,
while the generated `analyzer_min_vocab = 1` config indexes 18 terms and
retrieves correctly. WikiKB keeps generating lexcat.toml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@darthtrevino
fix(runtime): stop concurrent extraction from unlinking a published r…
8bca872 
…untime

extractLexcatArtifact published the freshly extracted runtime with an
rmSync of the install directory followed by a rename into its place. Two
`wkb` processes extracting at once each land in their own .extract-*
scratch directory, so the extraction itself was safe, but the publish
step was not: the loser of the race deleted the winner's already-verified
install directory before renaming its own copy over it. A sibling holding
a path into that directory then spawned a binary that no longer existed
and failed with ENOENT.

Publish with a bare rename instead. Renaming onto a populated directory
fails with ENOTEMPTY on POSIX and EEXIST/EPERM on Windows, which is the
signal that another process got there first; when the runtime already in
place matches the pinned digest it is adopted as-is rather than replaced.
A cached directory that is present but does not match is swapped aside
and retried, so a corrupt cache still self-heals.

This pattern was inherited from the SOMA integration, where the model and
preset setup made both processes reach the publish step far enough apart
to hide it. LexCAT starts fast enough that the window is actually hit.

The regression test raced only two processes and reproduced this about a
quarter of the time. Four racers reproduce it reliably; the scratch
assertion now also covers the .stale-* directories the retry path makes.

Verified on Linux: 2/8 failures before, 0/12 after, full offline suite
44/44.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@darthtrevino
Adopt LexCAT v0.0.14
074dc95 
Re-vendor all five runtimes and drop the two workarounds v0.0.14 retires.

The generated lexcat.toml is gone. It existed only to set
analyzer_min_vocab = 1, because the upstream default of 2 pruned every
term confined to a single chunk and made rare identifiers silently
unsearchable. v0.0.14 defaults the floor to 1, so the explicit pin is now
a no-op: an identical 66-chunk corpus built with no config yields 8 terms
and 0 hits on 0.0.13 versus 22 terms and 1 hit on 0.0.14.

Indexing now passes --json to build and sync and reads the reported
counts as fields instead of regex-matching them out of prose. The guard
also checks the term count, not just the chunk count: an index can hold
every chunk and still be unsearchable if the analyzer kept no terms, and
neither that nor an empty corpus is an error to LexCAT -- both build and
query at exit 0.

Dropping the config file cannot silently regress an operator-supplied
WIKIKB_LEXCAT_BIN older than v0.0.14, because --json is itself a v0.0.14
flag: 0.0.13 rejects it with exit 2 and the run fails loudly.

The index cache key is unchanged. runtimeCompatibility() already keys on
the runtime version, so every cached index is rebuilt on upgrade anyway.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@darthtrevino
Fix CI regressions introduced by the LexCAT migration
c9f1f65 
Two jobs this branch added were failing, both from bugs this branch
introduced rather than pre-existing issues on main.

Workflows and shell scripts: actionlint runs shellcheck over each 
un:
block, and the JS template literals inside the single-quoted
ode -e
programs tripped SC2016 ("expressions don't expand in single quotes").
Shellcheck cannot tell a shell parameter expansion from a JS one. Replace
every template literal in those blocks with string concatenation.

LexCAT Windows x64: 	ools/wikikb-local/wkb is an extensionless bash
script with no Windows shim, so Windows cannot spawn it and spawnSync
returns status null. The two vendored tests were gated only on artifact
presence, and a win32/x64 artifact does exist, so they ran and failed.
main gates the equivalent SOMA tests on the platform for this reason;
restore that gating. The Windows job still exercises the vendored runtime
directly through the LexCAT CLI (digest, --version, build/sync --json,
query), so the platform keeps real coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@darthtrevino darthtrevino changed the title Replace SOMA with the LexCAT lexical runtime (v0.0.13) Replace SOMA with the LexCAT lexical runtime (v0.0.14) Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@darthtrevino