ci-runner is the Windows host controller and disposable Linux worker image
for a small, demand-scaled GitHub Actions fleet. Its purpose is to move eligible
private-repository CI from paid GitHub-hosted runners onto melo-desk-001 and
melo-lap-001 without creating a second CI contract. There is no routing
policy: an eligible private job names the governed fleet label
melodic-ubuntu-24.04-x64 as a literal, and GitHub queues it for that label.
Both hosts passed their acceptance gates and serve every private job that names
the fleet label, which is every eligible private job that does not hold a
declared hosted exception. The former
Compose/restart-in-place implementation is retired and its files are deleted;
the only production credential entry point is
ci-runner secret import --file PATH.
flowchart LR
W["Workflow job"] -->|"names melodic-ubuntu-24.04-x64"| S["Host-owned scale sets"]
W -->|"names an approved hosted label"| H["GitHub-hosted ubuntu-24.04"]
S --> C1["Windows controller\nmelo-desk-001"]
S --> C2["Windows controller\nmelo-lap-001"]
C1 --> D1["Fresh one-job Linux containers"]
C2 --> D2["Fresh one-job Linux containers"]
Each host owns an independent scale-set ID and listener session. Organization
hosts advertise the same workflow label, so GitHub can distribute work and
reassign a job before acquisition if one host disappears. The controller uses
GitHub's official Runner Scale Set Client
outside Kubernetes and scales from its authoritative TotalAssignedJobs
statistic.
Every admitted job gets a new container from the digest-pinned official
actions/actions-runner image. The controller streams the one-job JIT payload
over attached stdin; the entrypoint exposes it to the official runner through
the documented ACTIONS_RUNNER_INPUT_JITCONFIG input. The payload is absent
from Docker's persistent config and docker inspect. The container has no host
mount, Docker socket, device, GPU, persistent home, work directory, temp, or
tool cache, and is removed after terminal diagnostics are copied out.
The Windows process is the only control plane. GitHub App keys are protected
with current-user DPAPI and never enter worker containers. The controller talks
only to the fixed local Docker Engine endpoint and requires a Linux/amd64
engine; DOCKER_HOST, TLS, and API-version environment overrides are ignored.
There is no selector and no fallback. ci-perf Phase 7 retired the central
select-runner reusable workflow in
melodic-software/ci-workflows
(ci-workflows#569, merged as 541ee4e90d12d77a90a3ddd72a3af9bc78634ea7,
released as v0.23.0), and melodic-software/standards#556 (merged as
771a796628f325c3c418c7b397d09fb7211e2972) removed the selector grammar from
the runner-policy component. The three policies this section used to document,
hosted-only, prefer-self-hosted and self-hosted-only, no longer exist. The
CI_RUNNER_POLICY organization variable that chose between them still exists at
self-hosted-only and reads nowhere: the selector that consumed it is deleted,
and its removal from the Pulumi program is decided pending the owner's Phase 7
step 5 apply, so changing it changes nothing. github-iac OrgCiRouting.cs still
declares it, unprotected ahead of that apply.
An eligible private-repository job names the governed fleet label
melodic-ubuntu-24.04-x64 as a literal in its own runs-on. GitHub queues the
job for that label, and this controller supplies runners inside its governed
name and label namespaces exactly as before. What changed is upstream of the
controller: nothing decides at run time where a job goes. The
standards runner policy
admits that literal under its managed-literal routing kind and fails a job
that names anything else without a declared exception.
A busy runner does not divert work. GitHub queues the job until matching capacity is available, which was already the behavior under the retired liveness rule and is now structural rather than policy-dependent.
There is no consumer-side recovery from an unavailable fleet. The audited
github-iac routing-control procedure is deleted with the workflow that
implemented it (github-iac#453, merged as
4c5937e6b6067552d11b87abef04620191dda503), and nothing replaces it. Affected
jobs queue on the fleet label until a host returns.
The exceptions mechanism does not cover this case, and reaching for it
will fail. A job that needs hosted capacity declares an entry under exceptions
in its repository's own .github/runner-policy.json, keyed
<workflow path>#<jobId>, carrying a justification and a reason that must
be a member of the closed hostedExceptionReasons set: dependabot,
docker-socket, job-container, privileged-control-plane, publication,
service-container, windows. None of those expresses "the fleet is
down", and a reason outside the set is rejected by the analyzer. Granting a
fleet-outage escape would mean adding a member to that set, which is a governed
change to melodic-software/standards and its synced policy.json, not
something a consumer repository can do for itself.
(hosted-exception-required is the finding the analyzer raises when a required
entry is missing, not a key anyone writes.)
A rerun changes nothing about placement: there is no selector verdict to
recompute, so Re-run all jobs and a failed-job rerun are equivalent as far
as routing is concerned. Do not re-run a stale run on a superseded head SHA in a
pull request whose concurrency group key does not vary with the head; that
hazard is unrelated to routing and is recorded in github-iac
docs/topics/ci-perf/POSTURE.md under "Stale re-runs". A workflow_dispatch
still creates a separate run with different event and ref context and does not
recover the original pull-request check. GitHub documents the distinct
full and partial rerun
operations
and workflow_dispatch event
context.
The two-minute ubuntu-slim selector control job is gone with the selector, so
it no longer sits in front of a build. The decision record is github-iac ADR
0014, which supersedes ADR 0004. It is added by melodic-software/github-iac#466
and is not on that repository's main yet, so cite the pull request until it
merges.
Authoritative behavior:
ci-runner.exe is the only operator interface. With no subcommand it opens the
interactive menu; the same operations are available for automation:
ci-runner host
ci-runner host status [--json]
ci-runner host enable [--wait]
ci-runner host disable [--wait|--detach]
ci-runner host game [--wait|--detach]
ci-runner host doctor [--json] [--include-elevated]
ci-runner host logs [--follow|--job ID|--cleanup]
ci-runner host force-stop
ci-runner host controller restart
ci-runner host controller stop-for-update
ci-runner host reboot [--timeout DURATION] [--force] [--dry-run]
ci-runner secret import --file PATH
Modes are persisted separately from checked-in configuration:
enabledstarts Docker when policy allows, advertises capacity, and maintains configured warm workers;disabledadvertises zero, drains CI, and removes idle workers without touching unrelated Docker or WSL workloads;gamingdrains CI, stops Docker Desktop, shuts down every WSL distribution, verifies both are down, and remains down across logons.
No normal timeout implies force. Busy jobs finish naturally, including drains
longer than the warning threshold. force-stop is a separate destructive path
that inventories affected jobs and requires typed confirmation. Ctrl+C while
watching a drain detaches from the display; it does not cancel the drain.
host doctor is non-elevating by default. It reports the BitLocker check as
skipped while still running every ordinary diagnostic. --include-elevated
explicitly enables the full BitLocker verification and writes a warning before
the verifier can open an Administrator UAC prompt; use it only in an interactive
session where that prompt is expected.
host doctor also surfaces pending-OS-reboot state from the standard registry
signals (component servicing, session-manager pending file renames, and Windows
Update). Because updates auto-install but the host never auto-reboots while a
session exists, a pending reboot is expected operational state the operator
finishes during a deliberate drain window: the check is advisory, renders as
WARN, and never degrades the doctor exit code.
host reboot is the operator command that finishes that drain window: it
reuses the stop-for-update capacity-zero drain so desired mode is untouched,
then restarts the machine through trusted shutdown.exe /r. --timeout bounds
the drain wait; on expiry the controller keeps draining on its own. The command
refuses to restart after an unclean drain unless --force is passed.
--dry-run drains and reports without requesting a restart. The command does
not pass shutdown.exe /f; --force means "reboot after an unclean drain",
not "forcibly close applications".
The windowless ci-runner-controller.exe runs from a current-user logon task
because Docker Desktop is user-session software. A controller restart first
uses the authenticated control plane to bind the shutdown reservation to the
preflight PID, version, and job counts, then advertises zero capacity and lets
every assigned or active job finish. Only after every pool has two confirmed
zero-capacity observations, no active worker remains, and the message sessions,
worker runtime, and control server close successfully does the controller write
an ACL-hardened, atomic completion receipt bound to the authenticated request
ID, old PID, and exact version and emit its dedicated restart exit code. The CLI
waits on that exact process handle and requires both the exit code and matching
durable receipt before it can touch Task Scheduler.
The CLI then asks Task Scheduler to run the canonical ci-runner-fleet task
at most four times with exponential backoff. Those bounded retries close the
MultipleInstances=IgnoreNew race while Task Scheduler finishes the prior
instance. Success still requires the authenticated control plane to report a
different PID at the exact expected version. An unavailable initial control
plane, an ordinary controller failure, or a missing or mismatched receipt never
authorizes a task start.
The CLI invokes the existing task through a direct hidden schtasks.exe /Run
child process. It does not launch the controller directly, open another console
window, invoke a shell, request elevation or UAC, or terminate the draining
controller. Battery, resource admission, drain, Docker Desktop, and WSL policy
remain in the shared Go state machine.
The checked-in, nonsecret host YAML is owned by
melodic-software/provisioning and is
installed as %LOCALAPPDATA%\ci-runner\config.yaml. The strict parser rejects
unknown properties, unsupported schemas, invalid units, duplicate targets,
unsafe paths, inconsistent thresholds, and explicit null or blank target worker
resource sections and fields. YAML merge keys (<<) are rejected everywhere so
inheritance cannot bypass those checks.
Provisioning verifies ownership boundaries through the product rather than
parsing YAML itself. config validate --json returns the normalized
release.compatibilityManifest and paths (secrets, state, logs, and
diagnostics) contract. host status --json returns the authenticated live
controller's PID, exact version, phase, shutdown state, and job counts under
controller; provisioning requires a new nonzero PID and the requested version
before committing an install transaction.
Mutable local state is separate:
%LOCALAPPDATA%\ci-runner\
state\desired.json # user-owned mode and temporary capacity override
state\observed.json # controller heartbeat, pools, workers, problems, drain reason
state\jobs.json # exact job-to-artifact correlation
state\restart-completed.json # last authenticated restart completion receipt
secrets\ # current-user DPAPI-protected App keys
logs\controller\ # structured JSON Lines controller events
logs\workers\ # externally captured runner stdout/stderr
diagnostics\ # copied runner _diag and terminal cgroup evidence
Permanent capacity and threshold changes are YAML changes, never source-code changes. A menu capacity override is local state and can be reset to the checked-in value. Provisioning must not overwrite desired mode.
One diagnostics policy governs both copied runner stdout and compressed _diag
archives. maxFileSize, rawDiagnosticMaxInput, retention, total-cap, and
cleanup cadence are strict host configuration. Startup retention runs only
after every managed active or exited container has been adopted. The explicit
host logs --cleanup command first inventories the fixed local Docker endpoint
and refuses to run if that safety inventory is unavailable.
Optional OpenTelemetry export uses a reviewed telemetry: host configuration
block, with standard OTEL_* environment variables retained for legacy and
exporter-specific settings. It is disabled without explicit configuration and emits controller reconcile
spans and low-cardinality fleet, capacity, worker, job, host-pressure, gate, and
lifecycle metrics. Exporter failures are locally logged and cannot change
capacity, drain workers, cancel jobs, or stop the controller. Runner names,
container IDs, job IDs, credentials, and arbitrary error text never become
metric attributes. See OpenTelemetry observability for
the exact enablement, metric, cancellation, and failure-isolation contracts.
Default worker parity is 2 CPU, 8 GiB memory, no additional swap, 4096 PIDs, and no devices. Admission also honors configurable host memory/CPU thresholds, hysteresis, global worker limits, and laptop AC-only policy. Active workers are never killed to reclaim resources.
An enabled pool can keep one excess healthy idle runner as bounded burst inventory instead of advertising zero capacity for the whole pool to retire it. All pools share the host-wide advertised-capacity budget; when no safe slot is available, the excess runner falls back to quiescence. The next admitted job naturally consumes retained ephemeral inventory. Larger downscales and explicit zero-capacity modes retain the two-poll quiescence requirement before exact runner deregistration, so assignment races still fail closed.
While a GitHub listener poll is open, the controller refreshes managed worker inventory on the normal reconciliation cadence. A completed ephemeral container cancels the stale poll immediately so authoritative assignments can start replacement workers without waiting for the listener timeout.
Targets may optionally override individual fields from the global
resources.worker profile. Every omitted field inherits the global value; the
effective profile must still satisfy the same CPU, memory, total memory-plus-swap,
and PID validation. For example, this relevant excerpt keeps the ordinary pool
at 2 CPU/8 GiB while giving a CodeQL pool a larger profile:
schemaVersion: 2
resources:
maximumConcurrentWorkers: 5
minimumAvailableMemoryPercent: 25
memoryCapacityIncreaseMarginPercent: 25
worker:
cpus: 2
memory: 8GiB
memorySwap: 8GiB
pids: 4096
github:
targets:
- id: ordinary
# The remaining required target identity and capacity fields are omitted.
- id: codeql
resources:
worker:
cpus: 4
memory: 24GiB
memorySwap: 24GiB
pids: 8192Host admission computes the byte headroom above the physical-memory floor and
allocates prospective starts in target-priority order, charging each start's
effective memory profile. A larger target that does not fit is skipped so a
smaller eligible target can still run. Successful mixed-profile starts are also
reserved by their exact memory limits against fresh-but-potentially-stale host
observations for the rest of that reconciliation step. The reservation sum
saturates fail-closed, and maximumConcurrentWorkers remains the host-wide
ceiling. Insufficient memory headroom caps only prospective starts; it does not
activate the global resource gate or retire healthy existing capacity. Invalid
resource observations and sustained high CPU remain global gates. Target
profiles change only Docker CPU, memory, memory-plus-swap, and PID limits. They
cannot add the Docker socket, privileged mode, devices, or host mounts.
Listener capacity uses a per-pool memory Schmitt trigger. A decrease crosses the
raw worker-memory boundary immediately, preserving the fail-closed admission
contract. An increase must additionally clear
memoryCapacityIncreaseMarginPercent of that target's effective worker memory;
inside the band, the last advertised capacity is retained. The margin applies to
capacity growth only and is not charged as a permanent memory reservation.
Schema version 1 remains accepted without the margin and preserves the legacy
zero-margin behavior; the margin is required only by schema version 2.
The organization host App has only organization Self-hosted runners: write; each physical host gets a distinct private key. A separate observer App is read-only. Personal-repository host credentials are introduced only after the organization soak gate.
ci-runner secret import validates RSA PKCS#1/PKCS#8 PEM, BitLocker protection,
current-user DPAPI protection, and exact current-user/SYSTEM ACLs. Before it
reports success, it reads the protected destination back under the current
identity and verifies its fingerprint and import metadata. On Windows, the
import keeps the exact source file open with read and delete access while
withholding write/delete sharing. It rechecks that file identity at commit and
uses handle-bound deletion, so a moved, missing, or replacement pathname is
never deleted as though it were the imported PEM. If that identity check or
deletion is ambiguous, the verified DPAPI destination is retained and the
command fails with explicit manual-cleanup instructions. Failures before the
source-deletion commit leave the PEM untouched and roll back the destination
where possible. Any failed rollback or incomplete exclusive write that cannot
be removed is also reported as requiring manual cleanup and is never presented
as success.
The reported fingerprint is standard Base64 of the SHA-256 digest of the DER
public key. This is the exact format produced by GitHub's documented
openssl verification command,
so the CLI value can be compared directly with the fingerprint in the GitHub
App's settings.
Protected-secret schema v2 records that Base64 fingerprint. The loader remains backward compatible with schema v1 files, which stored the same SPKI SHA-256 digest as lowercase hexadecimal, and reports their fingerprint in canonical Base64 after validation. New imports never write the legacy representation.
Source cleanup is ordinary filesystem deletion. On Windows, deletion can fail
for reasons such as an incompatible open handle, a read-only file, or
insufficient access, and this product does not weaken those protections or
silently continue. The identity lock follows Microsoft's documented
CreateFile sharing contract,
where withholding delete sharing also prevents rename, and commit uses
SetFileInformationByHandle
with FileDispositionInfo, which requires delete access and deletes the opened
file object when its handle closes. Import is deliberately Windows-only; it
does not emulate this guarantee with a pathname-only unlink on other systems.
File deletion is not a forensic secure erase or a claim of media sanitization.
Use an appropriate sanitization process when the storage medium is reused or
disposed of, following NIST SP 800-88 Rev. 2.
The key is decrypted only in controller memory. Rotation overlaps old and new
keys until the new key can create a JIT runner, then revokes the old key.
The disposable worker necessarily contains its own decoded one-job runner credentials because that is how the stock GitHub runner connects and executes a job under one identity. Workflow code must be treated as able to inspect those ephemeral credentials. The enforced boundary is that no reusable App key, observer key, controller JWT, installation token, Docker control socket, or host filesystem enters the worker.
See the worker-image contract and GitHub's self-hosted-runner security guidance.
V1 is Linux x64 only. Versioned runtimes come from the same official setup
actions used on ubuntu-24.04; the image adds only the documented compatibility
baseline. GitHub-managed Actions caches continue to work across hosted and
self-hosted execution, but cache content is untrusted and must contain no
secrets.
These workloads stay GitHub-hosted:
- public repositories and fork pull requests;
- Windows jobs;
- service containers, job containers, Testcontainers, Docker actions, or jobs requiring a Docker socket;
- GPU/device/privileged workloads;
- broad cross-repository write/control-plane jobs;
- publication of this controller and worker image.
See GitHub dependency caching and Docker resource constraints.
Pull requests and pushes delegate Go quality to the exact-SHA-pinned reusable
workflow in melodic-software/ci-workflows. It runs the reviewed analyzer set
on native Linux and Windows, module tidiness and verification
plus
race-enabled tests on Linux,
ordinary tests on Windows, and authenticated
govulncheck analysis. The repository-local
Go build lane only cross-compiles the two Windows executables on Ubuntu; it does
not repeat those checks. Committed fuzz seeds run in ordinary tests, while the
weekly schedule and manual dispatch actively fuzz
both targets for 30 seconds each.
Other gates include Actionlint, Zizmor, strict configuration tests, and the live
worker-image verifier. Release tags first rerun the complete read-only gate
against the exact tagged source. Only the dependent publication job
receives the combined job-scoped contents:write, packages:write,
attestations:write, artifact-metadata:write, and id-token:write grant; the
later image-promotion job retains only contents:read and packages:write.
Worker provenance explicitly requests an organization artifact storage record,
and publication fails closed unless the pinned attestation action returns at
least one numeric storage-record ID.
Releases produce an immutable pair:
- versioned Windows ZIP and SHA-256 checksums;
- exact OCI worker image digest;
- controller and worker SBOM/provenance attestations;
- a compatibility manifest tying source SHA, controller, image, runner, Scale Set Client, Go, PowerShell, Buildx, BuildKit, and SBOM-generator pins together.
Dependencies and Actions are exact pins. Daily official-source drift evidence
opens an issue within 24 hours and hard-fails after 14 days; updates remain
reviewed and are never auto-merged. Deployment uses a versioned install
directory plus current junction; how many known-good pairs are retained is a
floor, stated once in the freshness policy.
Rollback order is: drain without killing work, restore the prior immutable pair,
restore the prior reusable-workflow SHA if needed, then use Re-run all jobs
for affected workflows. The first step used to be flipping routing to
hosted-only; there is no routing variable to flip and no consumer-side
substitute, so a fleet the rollback cannot restore means the affected jobs
queue until it is back. Plan the rollback window accordingly: the fleet's
availability is now the availability of every private job that routes to it,
which is all of them except the ones already holding a hosted exception. There
is no quick shortcut either, because widening the reason set is a governed
change to the standards runner policy and each affected repository still
needs its own per-job exceptions entry and a runs-on change before anything
routes differently.
A completed one-job worker's captured runner log can contain an
ERR BrokerServer line carrying a TaskCanceledException when the stock runner
cancels its broker long-poll at teardown. This is expected teardown noise, not a
failed job or worker; see
Cancellation and runner shutdown noise.