Skip to content

Implement SQLite-backed FileCache for ontology lookup cache - #506

Open
RichardBruskiewich wants to merge 11 commits into
mainfrom
file-cache-for-ontology-lookup
Open

Implement SQLite-backed FileCache for ontology lookup cache#506
RichardBruskiewich wants to merge 11 commits into
mainfrom
file-cache-for-ontology-lookup

Conversation

@RichardBruskiewich

@RichardBruskiewich RichardBruskiewich commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

This ontology lookup module is meant to support Translator ingests (e.g. like HMDB) that have conceptual annotation of data from primary knowledge sources which are text string rather than ontology term based .

The file-cache-for-ontology-lookup development branch in this PR extends the ontology-lookup-utility branch of PR #505 - that wraps the Translator phase 2 built SRI RENCI team built Name Resolution Service ("NRS") endpoint - with a SQLlite implementation of a backend database cache of local concept name string to ontology term resolution.

The purpose of the file cache is obviously to reduce duplicate accesses of the NRS API web service thus, enhance performance.

(Note that the code prior to commit 7e1163b is based on the EBI Ontology Lookup Service but this earlier implementation was subsequently ported away from the OLS, and rather, towards the NRS)

Initial feedback from Sierra:

  • Reinventing the wheel(?): should OakLib be used?
  • Would the NRS be accessed too much during ingests?
  • Could simple Koza mapping files be pre-computed instead of a (preloaded) SQLlite cache (the latter being a relatively heavyweight dependency)?

Initial response from RB:

  • The PR code is a simple NRS wrapper, with simple API, coded and working.
  • NRS also provides term scoring and Biolink Model category assignment, useful for ad hoc concept string mappings
  • The NRS is only accessed once for every unique string within any ingest using the library; terms are otherwise cached into the (SQLlite) file cache
  • Substitution of a (Koza) map table/file solution (for the SQL) doesn't remove the need for pre-computed original (NRS or source ontology) term lookup (some new alternate code still needed, plus scripts to pre-calculate them)
  • OakLib does feel a bit overkill in terms of complexity relative to the simple wrapping of the NRS endpoint for its narrow use case; steep learning curve to be climbed leverage OakLib here (somebody more proficient than this PR submitter would likely be in a better position to tackle this).
  • OakLib is as heavyweight a dependency as SQLlite; If OakLib is to be used, should a separate repository be created for Translator Ingest ETL support scripts, such as (Koza) map loading scripts, to avoid addition of OakLib to Translator Ingest dependencies? Do we otherwise set up some kind of partitioning of 'uv' dependencies (in the pyproject.toml) plus special Makefile targets, to keep script dependencies and main ingest dependencies separate(?)
  • Use of OakLib doesn't remove internet access to ontology files (or endpoints) but modifies how (e.g. web endpoints versus OBO file downloads?) and change when it is done (aim for pre-computed cache tables either way?)

Summary

  • Implements FileCache (cache/file_cache.py): a dictionary-like, SQLite-backed cache of ontology term lookups. One database file per ontology under cache/db/, indexed by (query, only_taxa).
  • only_taxa is normalized to '' internally so it participates safely in the composite primary key (SQLite treats NULL as never equal to itself in unique/PK indexes).
  • Entries store either a resolved term (JSON) or an explicit "unresolved" tombstone (NULL term_json), so a query the NRS previously had no match for is no longer silently mistaken for "never looked up" and re-queried every run.
  • cache/__init__.py: removed the dead FileCache stub, wired cache_lookup to the new store, and threaded only_taxa through contains/retrieve/save end-to-end (previously dropped).
  • cache/db/*.sqlite3 is committed to the repo (not gitignored) so the cache is shared across users and the production pipeline rather than rebuilt per checkout; seeded here with one UBERON entry (Placenta -> UBERON:0001987).

Tests

  • tests/test_file_cache.py: offline functional tests covering round trip, upsert, unresolved-vs-never-cached, ontology partitioning, only_taxa partitioning, cross-instance disk persistence, and the dunder protocol (in/[]).
  • pytest tests/test_file_cache.py -> 8 passed.
  • pytest --doctest-modules file_cache.py -> passed.

The SQLlite portion of the PR was largely inspired and documented with Claude Code, using detailed specification prompts from @RichardBruskiewich)

…various real world datasets (might be worth externalizing the string mappings into a file?)
…efinement to properly process GO data sample ("apoptosis") plus need better unit test of expectations
…efinement to accurately resolve some test cases
- FileCache (cache/file_cache.py): dictionary-like contains/retrieve/save
  (plus __contains__/__getitem__/__setitem__ sugar), backed by one SQLite
  database per ontology under cache/db/, indexed by (query, only_taxa).
  only_taxa is normalized to '' internally so it participates safely in
  the composite primary key (SQLite treats NULL as never equal to itself
  in unique/PK indexes).
- Entries store either a resolved term (JSON) or an explicit "unresolved"
  tombstone (NULL term_json), so a query that NRS previously had no match
  for is no longer silently mistaken for "never looked up" and re-queried
  every run.
- cache/__init__.py: removed the dead FileCache stub, wired cache_lookup
  to the new store, and threaded only_taxa through contains/retrieve/save
  end-to-end (previously dropped).
- cache/db/*.sqlite3 is committed to the repo (not gitignored) so the
  cache is shared across users and the production pipeline rather than
  rebuilt from scratch per checkout; seeded here with one UBERON entry.
- tests/test_file_cache.py: offline functional tests covering round trip,
  upsert, unresolved-vs-never-cached, ontology partitioning, only_taxa
  partitioning, cross-instance disk persistence, and the dunder protocol.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant