Implement SQLite-backed FileCache for ontology lookup cache - #506
Open
RichardBruskiewich wants to merge 11 commits into
Open
Implement SQLite-backed FileCache for ontology lookup cache#506RichardBruskiewich wants to merge 11 commits into
RichardBruskiewich wants to merge 11 commits into
Conversation
…various real world datasets (might be worth externalizing the string mappings into a file?)
…efinement to properly process GO data sample ("apoptosis") plus need better unit test of expectations
…efinement to accurately resolve some test cases
… resolution service.
- FileCache (cache/file_cache.py): dictionary-like contains/retrieve/save (plus __contains__/__getitem__/__setitem__ sugar), backed by one SQLite database per ontology under cache/db/, indexed by (query, only_taxa). only_taxa is normalized to '' internally so it participates safely in the composite primary key (SQLite treats NULL as never equal to itself in unique/PK indexes). - Entries store either a resolved term (JSON) or an explicit "unresolved" tombstone (NULL term_json), so a query that NRS previously had no match for is no longer silently mistaken for "never looked up" and re-queried every run. - cache/__init__.py: removed the dead FileCache stub, wired cache_lookup to the new store, and threaded only_taxa through contains/retrieve/save end-to-end (previously dropped). - cache/db/*.sqlite3 is committed to the repo (not gitignored) so the cache is shared across users and the production pipeline rather than rebuilt from scratch per checkout; seeded here with one UBERON entry. - tests/test_file_cache.py: offline functional tests covering round trip, upsert, unresolved-vs-never-cached, ontology partitioning, only_taxa partitioning, cross-instance disk persistence, and the dunder protocol.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This ontology lookup module is meant to support Translator ingests (e.g. like HMDB) that have conceptual annotation of data from primary knowledge sources which are text string rather than ontology term based .
The file-cache-for-ontology-lookup development branch in this PR extends the ontology-lookup-utility branch of PR #505 - that wraps the Translator phase 2 built SRI RENCI team built Name Resolution Service ("NRS") endpoint - with a SQLlite implementation of a backend database cache of local concept name string to ontology term resolution.
The purpose of the file cache is obviously to reduce duplicate accesses of the NRS API web service thus, enhance performance.
(Note that the code prior to commit 7e1163b is based on the EBI Ontology Lookup Service but this earlier implementation was subsequently ported away from the OLS, and rather, towards the NRS)
Initial feedback from Sierra:
Initial response from RB:
Summary
FileCache(cache/file_cache.py): a dictionary-like, SQLite-backed cache of ontology term lookups. One database file per ontology undercache/db/, indexed by(query, only_taxa).only_taxais normalized to''internally so it participates safely in the composite primary key (SQLite treatsNULLas never equal to itself in unique/PK indexes).NULLterm_json), so a query the NRS previously had no match for is no longer silently mistaken for "never looked up" and re-queried every run.cache/__init__.py: removed the deadFileCachestub, wiredcache_lookupto the new store, and threadedonly_taxathroughcontains/retrieve/saveend-to-end (previously dropped).cache/db/*.sqlite3is committed to the repo (not gitignored) so the cache is shared across users and the production pipeline rather than rebuilt per checkout; seeded here with oneUBERONentry (Placenta->UBERON:0001987).Tests
tests/test_file_cache.py: offline functional tests covering round trip, upsert, unresolved-vs-never-cached, ontology partitioning,only_taxapartitioning, cross-instance disk persistence, and the dunder protocol (in/[]).pytest tests/test_file_cache.py-> 8 passed.pytest --doctest-modules file_cache.py-> passed.The SQLlite portion of the PR was largely inspired and documented with Claude Code, using detailed specification prompts from @RichardBruskiewich)