Two views of the NCATS Biomedical Data Translator platform, built from the same set of components:
- The dashboard — a single self-contained HTML page with one row per component, showing which version each environment is running, where that version was read from, and where the environments disagree.
- The diagram — Graphviz pictures of how data flows between components and which services call each other.
They are two commands over two sources that are converging: the dashboard reads
components/*.yaml, which are committed here; the diagram still reads a Google
Sheet. Merging them is
issue #19.
The platform comprises many components maintained by different teams, and both views exist to make the whole thing visible at once — to people inside the project who need to see how the pieces fit together, to performance work that needs a map of where the bottlenecks might be (usefully combined with our OpenTelemetry data), and to people outside the project trying to understand how Translator works.
Requires Python ≥ 3.11 and uv — and for the dashboard, nothing else. No credentials, no Graphviz, no spreadsheet access:
uv sync # first-time setup; creates .venv/
uv run sync-components # follow what components/*.yaml points at -> data/sync/
uv run build-dashboard # compile that into a page -> data/dashboard/
open data/dashboard/index.htmlThat takes about ten seconds of network time and reaches only public services.
Everything it fetches is cached under data/, which is gitignored, so nothing
you run can dirty the repository.
The Graphviz system package (brew install graphviz,
or apt-get install graphviz), and read access to the Google Sheet the
component list still lives in — copy env.default to .env and
fill in GOOGLE_SHEET_ID.
uv run generate-diagram --google-sheet # download the sheet, then render
uv run generate-diagram # use the cached data/components.csv
uv run generate-diagram --all # include non-refactor components
uv run generate-diagram --google-sheet --format pdf
uv run generate-diagram --google-sheet --layer-column TierThe default view filters to components active in the current refactor ("Continues into Refactor" and "New in Refactor"), so the picture stays focused on what is currently relevant.
| If you want to | Look at |
|---|---|
| change what a component records | components/<id>.yaml, and schema/component.schema.json for the field reference |
| change the dashboard's row order or its stage descriptions | config/flow-steps.yaml |
| change a team's colour | config/owner-colors.csv, with the four constraints in docs/owner-colours.md |
| change what a published dashboard withholds | config/privacy.yaml |
| change the code | src/translator_diagram/, with the module map in AGENTS.md |
| know why something is the way it is | AGENTS.md for the working agreements, src/translator_diagram/CLAUDE.md for the code — particularly Things that look wrong but aren't |
| see what is planned | the issue tracker and FUTURE.md |
The first four are data files, editable by anyone who knows the platform without opening a Python module. That is deliberate, and worth keeping.
The canonical source of truth is a world-readable Google Sheet. Its ID is
stored in .env (gitignored; never committed). Copy env.default
to .env and fill it in:
GOOGLE_SHEET_ID=<paste the sheet ID here>Run with --google-sheet to download the latest CSV export into data/ and
use it immediately. The downloaded file is also gitignored.
The .env in the working directory, or the nearest one above it, is the one
that is read — so run the tool from your checkout, or set GOOGLE_SHEET_ID in
the environment.
The sheet must have an id column; everything else is optional (order does
not matter, and unknown columns are ignored). A CSV with no id column at all
is an error rather than an empty diagram — that is what pointing --sheet-gid
at the wrong tab looks like.
| Column | Description |
|---|---|
id |
Unique machine-readable identifier (kebab-case preferred) |
Name |
Human-readable display name shown in the diagram |
Owner |
Team that owns the component; controls node colour |
URL |
Link to the component's repo or docs; makes the node clickable in SVG output. Must be http:// or https:// — anything else is dropped with a warning |
Component in ITRB |
ITRB category (informational only) |
Refactor status |
Lifecycle status — see filtering below |
Gets results from |
Comma-separated IDs this component receives data from |
Calls |
Comma-separated IDs this component makes optional API calls to |
Externals |
Entities outside the diagram — <Name feeds in, >Name receives out. A name with neither prefix has no direction to draw and is dropped with a warning |
Part of |
Groups this component into a named cluster box |
Hosted at |
Deployment location; ITRB is the default and shown as nothing |
Ubiquitous |
TRUE to render this component as a per-caller clone (see below) |
Hide |
TRUE to suppress the component entirely: not as a ghost node, and not in components.json either |
Notes |
Free-text notes; shown as an SVG hover tooltip, not in the diagram |
IDs are matched case-insensitively; a case mismatch is a warning. An unknown ID, a duplicate ID, and two things that would end up sharing one SVG id (see SVG output and links) are hard errors that stop the run.
Prefix any ID in Gets results from or Calls with ~ to mark it as
planned but not yet implemented:
Gets results from: nodenorm-es, ~new-service
Calls: ars, ~future-api
Planned edges render in red; implemented edges render in black.
Cross-cutting infrastructure that nearly every component depends on
(telemetry, name resolution, logging…) creates long converging edges in
the diagram that obscure the real data-flow structure. Marking such a
component TRUE in the Ubiquitous column renders it as a small copy
next to each caller instead of as a single central node — the underlying
data stays normalised, only the visual layout duplicates. Jaeger (OTel)
is the canonical example.
All outputs go to data/ (gitignored) by default.
| File | Always? | Description |
|---|---|---|
data/diagram.png |
yes | Main shareable diagram |
data/diagram.dot |
yes | Graphviz source — useful for debugging or tweaking |
data/components.json |
yes | Every parsed component except the hidden ones (all statuses, not status-filtered) |
data/diagram_owners.png |
default | Owner-colour legend |
data/diagram_legend.png |
default | Edge-style legend |
data/diagram.pdf |
--format pdf |
Vector format for presentations |
data/diagram.svg |
--format svg |
Vector format for web embedding |
data/diagram_<layer>.png |
--layer-column |
One sub-figure per distinct column value |
data/diagram_<layer>.dot |
--layer-column |
Graphviz source for each sub-figure |
The two legend files are written separately by default; --no-split-legends
embeds them in the main diagram instead.
<layer> is the layer's value with punctuation folded to _. Two values that
fold to the same stem (Tier 1 and Tier-1) would overwrite each other, so
the second gets a _2 suffix and a warning.
components/<id>.yaml records one file per component: what it is (owner,
refactor status, layer, where it runs), its identifier in each of the naming
spaces Translator uses (GitHub repo, Helm chart, infores CURIE, wiki page),
ITRB's own app and group, links to its repositories and documentation, and
the connections: edges between components.
schema/component.schema.json is the field
reference; tests/test_component_files.py validates every file against it, and
tests/test_components.py covers the parser.
These files are the dashboard's only input, and they are committed — which is why the dashboard needs no credentials to run.
unknown.yaml is the holding pen for identifiers seen in the
platform that no component file claims yet — currently the OpenTelemetry
service names that could not be attributed. Entries leave it by being promoted
into a component file or confirmed out of use.
The dashboard reads these files; the diagram does not.
build-dashboard follows their pointers, but generate-diagram still loads
the sheet CSV — switching it over is
issue #19.
The case for the format is in
docs/component-metadata.md; the survey of what
each upstream source actually offers is in
docs/metadata-sources.md, and
docs/examples/name-lookup-enriched.yaml
shows what a fetcher builds from one of them.
A single self-contained HTML page with one row per component, in stages — data coming in at the top, the people who use it at the bottom — and a column per environment showing the deployed URL and the version running there.
uv run sync-components # follow the pointers, cache into data/sync/
uv run build-dashboard # compile into data/dashboard/
open data/dashboard/index.html
# Everything, including what a published build withholds. Local use only.
uv run build-dashboard --include-privateThe two steps are split because fetching is slow and rendering is iterated on:
one sync serves any number of rebuilds. sync-components re-fetches only what
is older than --max-age (15 minutes by default); --force ignores the cache.
A service being down is recorded rather than fatal, so the run still succeeds
and the page shows what was reachable.
It exists to test whether the metadata in components/ is worth keeping. Two
things it reports are the answer:
- Where each version came from. A version is read from the live OpenAPI
document where possible, then
/status, then the registered SmartAPI copy, then the Helm chart — and each cell says which, so "is OpenAPI a good enough source?" has a number rather than an opinion. Both live sources come before the registry, because a registration is a copy somebody filed by hand. - Which environments disagree. A version in the minority for its component is tinted, so drift between dev, ci, test and prod is visible without reading every cell.
- Which release each environment is running. The Repository column lists a
component's newest GitHub releases, plus any older release an environment is
still on, each tag linking to its notes. A filled tag is deployed somewhere
on that row, so the number in the
prodcolumn has its changelog one click away.
build-dashboard withholds what config/privacy.yaml
names — today the tracing console, the internal test rig, and the container
image tags that would turn the version grid into a CVE-matching inventory. The
page says so in its footer, and --include-private builds the full picture for
local use.
Redaction is the default so that a forgotten flag costs information rather than publishing it, and the file is data rather than code so that the people who know which systems are sensitive can edit it without opening a Python module. An entry that matches nothing is an error: withholding nothing is the failure worth guarding against, so renaming a component without updating the policy stops the build.
This is about reach rather than secrecy. Everything the page shows is read from public services and this repository is public, so the policy hides nothing from someone who looks — it keeps a few things off an indexed page. See issue #7 for the public/private split this anticipates.
Release lists come from the GitHub API, which allows 60 calls an hour to an
unauthenticated address. That covers a sync of this repository twice over, but
if you are re-syncing with --force set GITHUB_TOKEN in the environment for
a higher limit. It is sent to api.github.com and to nothing else, and a
throttled fetch costs some release tags, not the run.
Every column sorts. Click a header to sort by it, click again to reverse, and a third click returns to stage order — which is also what the ✕ beside the order name in the filter bar does, and the only way back when a narrow window has hidden the column you sorted on. The order is always named there, and it travels in the URL with the filters. In stage order the rows are banded by stage — each band naming what that stage is for — and sorting by owner bands them by owner instead.
The stages, their order, and the components in each are
config/flow-steps.yaml, edited by hand. The
recorded dependency edges are too sparse to order twenty-six components: with
nothing recording that the UI calls Name Lookup, a computed order put Name
Lookup up beside the data sources. A written order is more honest than a
plausible-looking one derived from data that is missing.
Last updated is the newer of a component's latest GitHub release and its
SmartAPI registration, badged with which one it was. Thirteen of the
twenty-six components have neither and show a dash: they publish no releases
and are in no registry. Nothing here dates a deployment — see
FUTURE.md for the closest available proxy. The environment
columns sort by the age of the release running in that environment, which is
the sharpest version of the same question: prod is on 0.6.1, released
2024-10-04.
The page opens on every component. The dropdown beside the owner filter
narrows it: Environments disagree (seven of twenty-four today, and the rows
with the most to say), Any version known, and No version known — which is
the list of gaps in the metadata, and worth reading as a to-do list. The count
beside the filters always says how many of the total are showing.
Filters live in the URL, so a filtered view can be pasted to someone else and
arrives as you saw it. Output goes to the gitignored data/: index.html,
which inlines its own data and so works straight from disk, and
overview.json, the same payload as a file for anything that wants to consume
it.
Owner-to-colour mappings live in
config/owner-colors.csv (two columns: owner,
color). Edit that file to add a new owner, re-order the legend, or change a
colour — no Python edit required.
The tool looks for that file in the working directory or any directory above
it, and falls back to the copy the wheel carries, so an installed
generate-diagram has colours wherever it runs. That copy is the same file:
the build maps it in, rather than the repository holding a second one.
--owner-colors PATH overrides both.
New owners not listed there receive fallback colours automatically.
- Bold border — component is "New in Refactor"
- Normal border — component "Continues into Refactor"
| Style | Meaning |
|---|---|
| Solid black arrow B → A | B provides results to A ("Gets results from") |
| Solid red arrow B → A | Same, but planned / not yet implemented |
| Dashed black arrow A → B | A makes an optional API call to B ("Calls") |
| Dashed red arrow A → B | Same, but planned / not yet implemented |
Where a solid edge already connects two nodes, a dashed edge in the same
direction is suppressed — otherwise --concentrate would merge the two and
lose the solid style. For the same reason, listing an ID both plainly and with
a ~ prefix in one cell (Calls: foo, ~foo) draws only the implemented edge.
Entries in the Externals column become nodes for things outside the diagram's
scope, filled amber so they stand out against the component colours:
- Sources (
<Name, cylinder) — entry points, e.g. the upstream data stores that feed intokgx-storage-pipeline - Sinks (
>Name, double-border oval) — exit points, e.g. the human end-consumer who receives results from the UI
Components that are referenced by an active component but are themselves
outside the current filter (e.g. "Removed after Refactor") appear as gray
dashed boxes labelled (excluded). This keeps cross-boundary edges visible
without cluttering the main diagram.
In SVG output every component node carries:
- an
<a xlink:href>wrapper when the row has aURL, so the node is a clickable link to that component's repo or docs — no JavaScript needed; - a hover tooltip with the owner, refactor status and notes;
- a stable
id, so a web page can address nodes directly fromcomponents.json. XML IDs can't contain spaces or slashes or start with a digit, so it is a sanitised form of the component'sid(ARS 2.0becomesars_2_0), andcomponents.jsoncarries the ids verbatim asnode_ids. That is a list: a ubiquitous component has no node of its own, only one clone per caller (ars__log,ara__log), and a component nothing references has none at all. Two things that would end up sharing one id — including a component whose id collides with a clone, an external entity, or thea_-prefixed wrapper graphviz puts around a linked node — are a validation error.
These attributes are inert in PNG output. They exist to support the planned interactive GitHub Pages view of this diagram — see below.
Run uv run generate-diagram --help for the authoritative list.
uv run generate-diagram [OPTIONS]
--input FILE Local CSV file [default: data/components.csv]
--google-sheet Download CSV from Google Sheet (reads
GOOGLE_SHEET_ID from .env in the current
directory or above) instead of --input
--sheet-gid INTEGER Google Sheet tab GID (0 = first tab) [default: 0]
--output-dir DIRECTORY Directory for output files [default: data]
--output-name TEXT Base filename for outputs [default: diagram]
--refactor-status TEXT Comma-separated Refactor status values to include
[default: "Continues into Refactor,New in Refactor"]
--all Include all components regardless of Refactor status
--format [pdf|svg] Additional output format beyond PNG (PNG is
always produced; can be repeated)
--direction [LR|TB] Graphviz layout direction [default: TB]
--concentrate/--no-concentrate Merge partially-parallel edges [default: off]
--split-legends/--no-split-legends
Write the legends as separate PNGs rather than
embedding them [default: split-legends]
--owner-colors FILE Owner-colour CSV to use instead of
config/owner-colors.csv
--layer-column TEXT Column to drive per-layer sub-figures, with
in-layer nodes bold-bordered and their
neighbours from other layers at normal
weight. In the current sheet this column is
named `Tier`.
--help Show this message and exit.
translator-diagram/
├── src/translator_diagram/
│ │ # shared by both stacks
│ ├── colors.py # Owner colours and the palette
│ ├── components.py # Reads components/<id>.yaml
│ ├── privacy.py # What a published dashboard withholds
│ │ # the diagram
│ ├── model.py # Component, index_by_id (one sheet row)
│ ├── naming.py # SVG ids and output filename stems
│ ├── loading.py # CSV parsing and the Google Sheet download
│ ├── validation.py # Reference and id checks
│ ├── render.py # The diagram and the per-layer sub-figures
│ ├── legend.py # The two legends
│ ├── export.py # components.json
│ ├── cli.py # generate-diagram
│ │ # the dashboard
│ ├── sync.py # Fetches what the component files point at
│ ├── flow.py # Data-flow depths, and the stage-order check
│ ├── dashboard.py # Version-source chain, drift, the rendered page
│ ├── dashboard_cli.py # sync-components and build-dashboard
│ ├── web/ # dashboard.css and dashboard.js, inlined into the page
│ └── CLAUDE.md # The module map and the non-obvious decisions
├── components/ # One YAML file per component — see docs/
│ └── CLAUDE.md # What a component file must contain
├── unknown.yaml # Identifiers no component file claims yet
├── config/ # The data files, edited by hand
│ ├── owner-colors.csv # Owner → fill colour
│ ├── flow-steps.yaml # The dashboard's stages, in page order
│ └── privacy.yaml # What a published build leaves out
├── schema/ # JSON Schema for components/*.yaml and unknown.yaml
├── docs/ # The metadata case, its research, owner colours
├── tests/ # One test file per module
├── .github/workflows/ # ci.yml (tests and lints), pages.yml (build + deploy)
├── pyproject.toml # uv/hatchling project metadata and dependencies
├── uv.lock # Pinned dependency versions
├── env.default # Template for .env (diagram only)
├── FUTURE.md # Ideas with their costs worked out
├── AGENTS.md # Working agreements, and where the rest is
└── data/ # Gitignored — every input and output goes here
├── sync/ # Cached upstream responses + manifest.json
├── dashboard/ # index.html and overview.json
├── components.csv # Downloaded from the Google Sheet
└── diagram.png # Rendered diagram (plus .dot, .json, .svg, .pdf)
Both commands work. The dashboard is also published: .github/workflows/ci.yml
runs the tests, lints and a build on every pull request, and
.github/workflows/pages.yml builds the page on every pull request as a
downloadable artifact while deploying only from main, a nightly schedule, or
an explicit manual run. A published build applies
config/privacy.yaml and carries noindex.
What is not built is the interactive view of the diagram — a static page
over the generated SVG and components.json. That is
issue #10,
and it waits on
issue #7:
this repository and its Pages site are public, so publishing a rendered diagram
publishes the component names, owners, statuses and dependency edges it
contains. Until that is settled nothing generated is committed, and data/
stays gitignored.
Worth knowing while it is being decided: the SVG carries more than the picture
shows. Every node's hover tooltip embeds its owner, refactor status and
Notes, and components.json carries every parsed column of every non-hidden
row — so "publish the diagram but not the details" means stripping tooltips and
JSON keys, not just labels.
The issue tracker is the live list. Ideas that started here:
- Report dependency cycles during validation — the validator catches unknown and duplicate IDs, but not a loop.
- Diff two runs and report what changed — e.g. before and after a sprint.
- Use the
Component in ITRBcolumn — it is parsed intoComponent.itrband nothing reads it. - Multiple sheet tabs.
--sheet-gidalready handles non-default tabs, so an--all-tabsmode could merge or overlay several views. Not filed as an issue: whether it is worth anything depends on how the sheet ends up structured, which is part of #7.
uv sync # first-time setup
uv run pytest # the test suite
uv run ruff check # Python lint
uv run rumdl check . # Markdown lintAll four run in CI on every pull request, along with a dashboard build. The
source is one module per subject under src/translator_diagram/, and tests/
has one file per module — a change to loading.py belongs in
tests/test_loading.py.
Two things are worth reading before a first change. AGENTS.md has
the module map, the rule about which module may import which (enforced by
tests/test_package_layout.py), and a Things that look wrong but aren't
section that exists because most of them were arrived at the hard way. And if
you change the dashboard, render it and look at it: the page has passed a full
test suite while visibly broken more than once, and AGENTS.md carries the
headless-Firefox recipe for checking it at several widths and in both themes.
MIT.