Skip to content

CLAUDE.md

Latest commit

 

History

History
325 lines (308 loc) · 23.3 KB

CLAUDE.md

File metadata and controls

325 lines (308 loc) · 23.3 KB
Raw

CLAUDE.md

Lithuanian Linked Open Data: re-runnable ETL pipelines (etl/) that regenerate RDF datasets (datasets/current/) from live open-data APIs on every run.

Commands

cd etl && make                 # everything: taxonomies → admin-units → seimas → legal-entities
make -C etl/<domain> all       # one domain (fetch is always fresh — FORCE prerequisite)
make -C etl/seimas photos      # opt-in: scrape official portraits from lrs.lt
make BASE=https://linkeddata.lt/   # prod base URI (default https://localhost:4443/, see etl/config.mk);
                               # committed datasets/current/ are base-RELATIVE (no @base) — identical
                               # for any BASE; the base is (re)applied at load/parse/validate time

etl/queries/run.sh <q.rq>      # SPARQL over ALL datasets loaded in-memory (~1M quads, -Xmx4g)
python3 etl/queries/render-examples.py   # regenerate etl/queries/EXAMPLES.md result tables

uv run --project etl/tools ltlod-reconcile <admin-units|persons> [--base …] --input … --output …

make sef                       # compile the client-side XSLT override (files/client.xsl) to a
                               # Saxon-JS SEF (files/client.xsl.sef.json, gitignored). Run once
                               # BEFORE the first `make up` (the compose mount needs the file to
                               # exist) and after every edit to files/client.xsl; then recreate
                               # the linkeddatahub container to reload. See "Client-side XSLT".
make up                        # deploy LinkedDataHub at https://localhost:4443/ (root Makefile;
                               # bootstraps secrets + server cert, then docker compose up -d)
make install                   # set up the dataspace via LDH CLI: make it public + PUT app/
                               # scaffolding (root + containers + taxonomy schemes) + install
                               # app/ns.ttl (1:N views) into the admin ontologies/namespace/
                               # doc; needs ../LinkedDataHub (LDH_HOME=…).
                               # Interactive, LinkedDataHub-Apps style: prompts for Base URL /
                               # cert / password / proxy, defaults = the local stack (Enter×4
                               # or `printf '\n\n\n\n' | make install`); enter another Base URL
                               # + owner cert to install onto any LDH instance
make load                      # bulk-load datasets/current/*/*.trig into fuseki-end-user TDB2;
                               # resolves the base-relative TriG against BASE_URI (.env) via riot
                               # before tdb2.tdbloader; ends with `make public` (anonymous read)
make down / make drop          # stop stack / wipe LDH runtime state (never datasets/current/)

Prerequisites: Docker (only for atomgraph/csv2rdf; no docker-compose), Apache Jena 5.6.0 (JENA_HOME — arq 6.x cannot output quad CONSTRUCT results at all: "No dataset writer for Turtle_pretty", --results=trig dropped; see apache/jena#4091), xsltproc, uv, make, curl. The make sef target additionally needs Node/npx (xslt3-he, the Saxon-JS compiler) and xmlstarlet.

Architecture

Every domain runs the same four stages (shared scripts in etl/lib/):

  1. fetch — full dump from live API (Spinta UAPI CSV export with select(...) incl. dereferenced parent keys, e.g. apskritis.adm_kodas; Seimas XML). Fails loudly on empty results. Cache dirs are gitignored.
  2. normalize — CSV → CSV2RDF identity transform (docker), XML → XSLT 1.0 (xsltproc), both emit source-shaped RDF with <{base}#column> properties.
  3. graphify — LDH-style quad CONSTRUCT { GRAPH ?graph {…} } mapping (mappings/*.rq, $base-parameterized) executed by arq; the CONSTRUCT mints ABSOLUTE IRIs in-model, then etl/lib/relativize.sh rewrites the output to base-RELATIVE IRIs with no @base → TriG. The .rq files are reusable verbatim as LinkedDataHub CSV imports. (ltlod-reconcile/photos take --base as the rdflib publicID and are relativized the same way; see the relative-TriG gotcha.)
  4. validate — riot --validate + every graph must have dct:title and foaf:primaryTopic on the graph URI (see etl/lib/validate.sh) + SHACL shapes per entity type (etl/shapes/<domain>.ttl, auto-selected by output dir, executed via etl/lib/shacl.sh; also run in CI on committed datasets by .github/workflows/shacl-validation.yml). Both validate.sh and shacl.sh take a base to resolve the relative IRIs (validate.sh pre-resolves to N-Quads).

Post-ETL: ltlod-reconcile matches entities to Wikidata (closed candidate sets via WDQS, exact label + parent disambiguation) and writes owl:sameAs + images into per-domain alignments.trig (same graph names as the entity docs, so they merge on load). Unmatched entities go to cache/unmatched*.csv, never force-matched. Emitted image URLs (the coat of arms P94 → foaf:img, other photos P18 → foaf:depiction, schema:logo from P154; and the lrs.lt portraits from the photos scraper → foaf:depiction) are https-normalized (WDQS returns http:// Commons URIs, which GitHub/browsers won't render) and HTTP-liveness-checked via ltlod_etl.images — dead images are skipped so they never enter the data (--no-image-check disables the check for offline/CI runs). The liveness check retries 429/503 with backoff (Commons rate-limits liveness bursts; a naive check treats the 429 as "dead" and strips live images en masse). P94 goes on foaf:img, not foaf:depiction, deliberately: foaf:img is rdfs:subPropertyOf foaf:depiction ("particularly representative"), and LDH's ac:image thumbnail selector ranks foaf:img > foaf:logo > foaf:depiction, picking only the first — so grid/card views (e.g. the frontpage AdminUnit grid) show the coat of arms deterministically, while the scenery photo stays a plain foaf:depiction on the entity page. Persons have no coat of arms but can still carry several foaf:depiction (P18 and the scraped lrs.lt portrait); showcase queries that render one image per row must collapse that (filter to one source, or GROUP BY + SAMPLE) — plain SELECT DISTINCT won't, since the rows differ in the image cell.

Conventions (load-bearing)

  • Entity per named graph: graph URI = document URI {base}{container}/{slug}/, entity = {graph}#this, secondary entities are #fragments. Slugs are natural keys from the source registry (AR codes, JAR codes, Seimas asmens_id) — any pipeline mints cross-links from bare foreign keys. Single source of truth: etl/URI-SCHEME.md — update it when adding containers.
  • LDH document hierarchy: ETL outputs are only dh:Item docs with sioc:has_container <its-container> — add both triples in any new mapping. Containers and taxonomy scheme docs come from app/ (one Turtle file per container, PUT via LDH CLI by make install); each carries an rdf:_1 <#select-children> → ldh:Object/ldh:ChildrenView block, without which LDH renders no children listing at all. app/persons.ttl swaps the stock ldh:ChildrenView for its own ldh:View + sp:Select (ac:GridMode) so the listing gets person filter/sort columns — see the next bullet.
  • 1:N entity views: cross-entity listings (county → municipalities, committee → members, party → nominees) are ldh:inverseView definitions in app/ns.ttl (the LDH namespace ontology, northwind-traders style): <property> ldh:inverseView <ldh:View> + spin:query → sp:Select with $about. LDH shows a view on every instance whose rdf:type matches the property's declared rdfs:range — exact type match, no subsumption, so view targeting relies on discriminating types in the data (schema:PoliticalParty for parties, cv:PublicOrganisation for the Seimas). app/import-ns.sh (called by make install) resets + POSTs the ontology into the admin ontologies/namespace/ document (served at {base}ns) and evicts the server-side ontology cache.
  • View filter/sort columns: LDH builds a view's facet pills and its sort dropdown from the BGP triples whose subject is the query's FIRST projected variable and whose object is a variable (ldh:RenderFacets and $var-predicates in LDH's client/block/view.xsl). Those variables must stay unprojected: LDH wraps the SELECT into DESCRIBE *, so every projected term becomes a rendered card/row — hence the "project a single entity type" rule in app/root.ttl. Consequences for authoring a view query:
    • the focus variable has to be the entity carrying the columns (?person = {doc}#this, never the dh:Item document — the attributes hang off #this);
    • a column is only a filter pill when its predicate is a plain URI (a property path still yields a sort option, labelled with the bare variable name);
    • column labels come from the {base}ns ontology, so every faceted predicate needs an rdfs:label in app/ns.ttl;
    • ORDER BY sets the initial sort; a second condition is the tie-breaker. Both are re-applied client-side over the (unordered) DESCRIBEd graph;
    • opening a pill makes LDH append a label lookup to the facet-value-count query that ARQ cannot index inside a GRAPH block (~104 s per pill → 500); files/client.xsl rewrites it — see the XSLT overrides below. Keep writing GRAPH ?graph in view queries. Person listings — the persons/ container view and the :CurrentMembers, :FormerMembers, :NominatedMembers, :MembersElectedHere ontology views — therefore project only ?person and carry foaf:givenName/foaf:familyName/ foaf:gender/ltlod:nominatedBy triples, sorted by family then given name.
  • XSLT overrides (files/): custom Saxon-JS templates that change how LDH renders, layered over the stock stylesheets — the linkeddatahub.com pattern. Three files, all mounted over the running image's ROOT/static for the end-user app only (admin untouched):
    • files/overrides.xsl — the shared module: suppresses the n-ary membership plumbing blocks (bs2:Row). Matching is property-centric via key(), never bare rdf:type, so a block is hidden only when it is a rendered target of the :Memberships 1:N view on this page — a membership whose org:member is key('resources', ac:absolute-path(ldh:request-uri()))'s foaf:primaryTopic; an interval/instant that is the object of org:memberDuring / time:hasBeginning|hasEnd (key('predicates-by-object', …)). Imported by both client.xsl and layout.xsl so the blocks are omitted server-side and client-side — server-only would still let CSR re-render them (flash); client-only lets the server render them first (flash).
    • files/client.xsl — imports client.xsl + overrides.xsl; overrides the stock ldh:bgp-value-counts facet label lookup: to label a filter pill's values LDH appends OPTIONAL { { ?value <7-way alt path> ?label } UNION { GRAPH ?g { … } } }, and ARQ does not resolve an alternative path through the quad indexes inside a GRAPH block — it walks the ~117k named graphs, so one pill took ~104 s (a 500 in the browser). The override emits the equivalent ?value ?labelProp ?label + FILTER (?labelProp IN (…)) (same predicates, same order, so SAMPLE(?label) is unchanged) — ~0.1 s per pill. It also adds the client-only :Memberships table tidy (restrict columns to role / organization / memberDuring, drop the anchor column, render the memberDuring cell as the interval's dct:title period text not a link). Compiled to the SEF; the view table is client-rendered so these needn't run server-side. Also overrides the stock geo-resources-string global param (from navigation.xsl — the default geo-map modal query) to add FILTER EXISTS { ?resource gsp:asWKT ?wkt }: the stock query DESCRIBE *s every geo:lat/long resource (~21k, incl. all point-only settlements) → ~21 MB → exceeds MAX_CONTENT_LENGTH (4 MB) → 502. The filter keeps only the ~606 admin units (the only geo resources with a WKT boundary) → ~3.1 MB. Scoped map views (:SubUnits, the frontpage counties map) use their own spin:query and are unaffected, so settlement points stay mappable there.
    • files/layout.xsl — imports base layout.xsl + overrides.xsl; repoints the client bootstrap (xhtml:Script → client-stylesheet) at our SEF, and replaces the stock bs2:Footer with the LTLOD one (own wordmark, dataset shortcuts, source attribution, licence; the developer entry points — SPARQL, example queries, dataset downloads, {base}ns — live there instead of on the frontpage). The footer keeps the stock markup contract, since app.css styles it structurally: .ldh-footer > .cols is a 1.4fr repeat(4, 1fr) grid, so exactly one .brand-col plus FOUR .col children, each a .ftitle followed by bare <a>s, then .legal with two space-between spans. Only the wordmark .mark deviates — restyled inline to the Lithuanian tricolour, class-supplied geometry kept. Footer-only because bs2:Footer is applied ONCE server-side (LDH layout.xsl) and never re-rendered by client.xsl — so it belongs here, not in overrides.xsl, and needs no SEF rebuild. Mounted at the end-user app's ac:stylesheet target static/xsl/layout.xsl; imports overrides.xsl from static/xsl/ (its own dir). Declare xmlns="http://www.w3.org/1999/xhtml" on xsl:stylesheet whenever this file emits literal result elements: LDH's own layout.xsl declares it, so its bare <div>s are XHTML; without it ours land in no namespace and the serialiser emits a stray xmlns="" on the block (HTML parsers ignore it, but namespace-sensitive XPath/XSLT matching on xhtml:* then misses those nodes). Wiring: make sef c14n's client.xsl+overrides.xsl and compiles the SEF against the pinned image's static/ tree; docker-compose.yml bind-mounts the four files (layout.xsl, overrides.xsl under static/xsl/; client.xsl, client.xsl.sef.json under static/com/ltlod/xsl/). Rebuild the SEF + recreate the container after editing any of them — docker compose restart linkeddatahub does NOT pick up an edited mount (it keeps serving the stylesheet compiled at the previous start); docker compose up -d --force-recreate linkeddatahub does, and then restart nginx/varnish per the 502 gotcha below. The :Memberships table replaces the suppressed blocks; it needs a readable period, so persons.rq puts a dct:title (e.g. "2024-11-14 – dabar") on each time:Interval — ldh:View/ac:TableMode is DESCRIBE-based, so date literals can't be view columns; the interval's dct:title surfaces in the org:memberDuring cell via object-label resolution.
  • Vocabulary cascade: W3C specs first → domain-specific third-party vocabs (EU SEMIC, OP authority tables, FOAF) → schema.org as general fallback → custom (http://linkeddata.lt/ns#) last. Rationale per domain: etl/ONTOLOGY-NOTES.md.
  • Change over time: n-ary org:Membership + org:memberDuring → time:Interval (W3C Time). No reification, no RDF-star. Current state = interval without time:hasEnd.
  • Docs in Lithuanian (README, EXAMPLES.md); code, code comments and CLAUDE.md in English. Python is uv-managed (etl/tools/).
  • Committed outputs live in datasets/current/; bulk regenerable files (settlements/streets TriG, ~59 MB) are gitignored.

Gotchas

  • BINDs inside OPTIONAL are evaluated bottom-up: a BIND referencing an outer variable (e.g. ?graph) silently unbinds and drops triples. Keep only triple patterns inside OPTIONAL; do URI construction after it, guarded with IF(BOUND(…), …, ?undef) (see etl/seimas/mappings/persons.rq).
  • CSV2RDF must run via docker — the local jar (../CSV2RDF/target) NPEs on modern JDKs (tested Java 25).
  • Jena shacl validate ignores TriG named graphs (validates the empty default graph → trivially conforms) and always exits 0 — etl/lib/shacl.sh flattens with riot --merge first and parses the --text report. Shapes must stay host-agnostic (path-suffix sh:patterns, never the base host).
  • Quad CONSTRUCT { GRAPH … } is an ARQ extension — works in arq CLI (its default syntax) and LinkedDataHub, not in strict SPARQL 1.1 engines.
  • Cross-graph queries use FROM <urn:x-arq:UnionGraph> (union of all named graphs as default graph). With only FROM given, GRAPH ?g {…} matches nothing.
  • Avoid per-row FILTER NOT EXISTS over the full dataset in ad-hoc integrity checks — quadratic, hangs for ~40 min on ~800k quads. Dump both URI sets with two SELECTs and comm -23 instead.
  • arq CONSTRUCT output ordering is nondeterministic across runs → large git diffs with no real changes. Compare with riot --output=nquads | sort when in doubt.
  • Spinta (get.data.gov.lt) is pre-alpha: model paths can drift; fetch scripts fail on unknown properties (HTTP 400) by design. Its :format/rdf output is nonstandard RDF/XML — always pull CSV instead.
  • Seimas API params: kadencijos_id (not p_kade_id); position rows may reference units absent from current feeds (dissolved commissions, the Board) — org-units mapping derives those from the members feed.
  • Switching BASE auto-invalidates static-CSV taxonomies via the cache/.base stamp in etl/taxonomies/Makefile (normalize outputs embed the base URI; fetched domains are immune — fetch is FORCE'd). Note make clean alone does NOT force the rebuild: missing cache/*.nt are intermediate files, which make skips when the .trig looks up to date.
  • make load bypasses LDH's HTTP API: it runs tdb2.tdbloader directly against the end-user TDB2 store via the one-off tdb-loader compose service (the atomgraph/fuseki image bundles the full Jena CLI inside the fuseki-server jar). Because the committed TriG is base-relative, the service first resolves it against BASE_URI (from .env, passed as -e BASE_URI=…) with riotcmd.riot --base=… --output=nquads, then loads the N-Quads (tdb2.tdbloader has no --base). Load is append-only — clean rebuild: make down && rm -rf fuseki/end-user && make up && make load. It stops fuseki-end-user first and removes the stale tdb.lock (lock PIDs are container-relative), then restarts the Varnish caches.
  • docker-compose.yml is a verbatim mirror of ../LinkedDataHub/docker-compose.yml (only build: . → image: atomgraph/linkeddatahub:5.6.0, because LTLOD pulls the published image and make sef/make up grep that line). Every LTLOD-specific delta lives in the committed docker-compose.override.yml (compose auto-merges it): the runtime image pin, TZ="Europe/Vilnius", ENABLE_WEBID_SIGNUP=false, and the tdb-loader bulk-load service (profiles: [ load ], so make up skips it and make load starts it via docker compose run). Re-sync from upstream by re-copying LDH's file and re-applying the single image-line edit — no other LTLOD edits touch the base.
  • Committed TriG is base-RELATIVE with no @base (base-agnostic: one dataset serves any deployment base). RDF stores only absolute IRIs, so the base must be reapplied on EVERY read — always pass --base to riot/arq/shacl (riot --base=$BASE …) or relative IRIs resolve to file://… garbage. The write path (etl/lib/relativize.sh) uses riot's STREAMING trig writer (--output, which relativizes against a prepended @base; --formatted does NOT) and strips the base header. Only base-internal IRIs relativize; external URIs (Wikidata, Commons, lrs.lt, EU tables, schema.org, mailto/tel) stay absolute. Round-trip proof: riot --base=$BASE --output=nquads file.trig | sort is base-independent.
  • Admin-unit coordinates live in *-geo.trig (ltlod-geo, from the AR spatial gra* models), merged on load like alignments.trig — the committed level *.trig carry no geo. LDH map mode (ac:MapMode) plots geo:lat+geo:long (WGS84 point marker) and/or gsp:asWKT (GeoSPARQL polygon). ⚠ EPSG:3346 (LKS-94) axis order is (Northing, Easting): the source WKT stores the ~6·10⁶ northing first, ~3–6·10⁵ easting second, so ltlod-geo feeds pyproj (easting, northing) under always_xy=True → (lon, lat). Every point is bounds-checked against Lithuania's bbox — a swapped axis lands far outside and aborts. Sanity check: Vilnius ≈ lat 54.6–54.7, lon 25.2–25.3.
  • Geometries are gsp:asWKT on #this, simplified, coarse levels only (ltlod-geo --simplify <metres>, set per level in admin-units/Makefile). The WKT must be bare WGS84 lon-lat (no leading <crs> URI — OpenLayers' ol.format.WKT can't parse one) typed http://www.opengis.net/ont/geosparql#wktLiteral; put it directly on #this (not a geo:hasGeometry node) so the map feature id is the unit URI. LDH does no client-side simplification (raw polygons are ~160 MB), hence Douglas–Peucker in ETL. No LTLOD XSL change needed — map.xsl ships via the imported stock client.xsl; the committed coarse *-geo.trig hold points + WKT, settlements stay point-only.
  • Fuseki ports are never published to the host — query via https://localhost:4443/sparql or from inside the network: docker compose exec linkeddatahub curl http://varnish-end-user/ds/.
  • LDH strips client-sent sioc:has_parent/sioc:has_container on PUT and manages the hierarchy itself (re-adds sioc:has_parent for dh:Container, adds dh:Item + sioc:has_container otherwise, plus dct:created/ acl:owner) — never put sioc triples in app/*.ttl. make install is idempotent: PUT replaces the whole named graph.
  • Public read access is class-based: make public (direct-to-fuseki) and make install (LDH CLI make-public.sh, works remotely) grant the same acl:accessToClass def:Root, dh:Container, dh:Item, nfo:FileDataObject — ETL documents match because mappings type them dh:Item/dh:Container. (Untyped docs would also pass: LDH's ACL query leaves $Type unbound when a document has no rdf:type, matching any acl:accessToClass — see AuthorizationFilter + aclQuery in LDH web.xml.)
  • COMPOSE_PROJECT_NAME=ltlod isolates container/volume names from other local LDH stacks, but ports 81/4443/5443 still clash — one stack at a time.
  • 502 on all public endpoints after restarting backend containers (fuseki, varnish): nginx resolves upstream container IPs at startup — restart nginx too. fuseki-end-user can be OOM-killed (exit 137) under memory pressure when other Docker workloads run; docker compose up -d fuseki-end-user revives it (LDH health recovers on its own).

Verification

After changing a mapping: rebuild the domain, check the reported graph count against source counts (10 counties / 60 municipalities / 584 elderships / 148 MPs), then run the link-integrity pattern from etl/queries/ (referenced #this targets vs foaf:primaryTopic set — must be 0 dangling) and eyeball one entity graph. python3 etl/queries/render-examples.py re-runs all example queries end-to-end. SHACL shapes describe current data — if make fails the shapes check after a mapping change, update etl/shapes/<domain>.ttl in the same commit.