Spyglass

A Python CLI tool for CPU and memory profiling of Lighthouse.

WARNING: This tool is a work-in-progress and largely vibecoded. There will be bugs

Screenshot

Features

• CPU profiling: via perf record with automatic flamegraph generation
• Memory profiling: via jemalloc heap dumps
• Epoch boundary isolation: capture profiles around specific epoch transitions
• Category-based analysis: configurable pattern matching to group samples by subsystem
• Custom flamegraph timeline: timeline of flamegraphs across an epoch boundary colour-coded by subsystem
• Comparison: side-by-side delta reports between two profiling runs
• Metrics scraping: Prometheus metrics captured at epoch boundaries for cache/timing analysis

Requirements

• Python 3.11+
• uv (recommended) or pip
• Linux with perf installed
• inferno (cargo install inferno)
• jeprof (from the jemalloc package) (for memory profiling)

Installation

# Install uv if you don't have it
# Arch: sudo pacman -S uv
# Other: curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone and install
git clone <repo-url> spyglass
cd spyglass
uv venv
uv pip install -e .

# Activate the virtual environment
source .venv/bin/activate    # bash/zsh
source .venv/bin/activate.fish  # fish

# Now `spyglass` is available directly
spyglass --version

Quick Start

If you've installed with uv pip install -e ., you can use spyglass directly. Otherwise, use python3 -m spyglass instead.

# Configure (at minimum you will need to add the path to your Lighthouse directory)
cp configs/config.example.toml configs/config.toml
vim configs/config.toml

# Full workflow: build + run
spyglass profile --mode cpu -n my-experiment

# Or step by step
spyglass build --mode cpu
spyglass run --mode cpu -n baseline

# Then analyze
spyglass analyze baseline --filter all

Commands

build

Builds Lighthouse with profiling instrumentation.

python3 -m spyglass build --mode cpu      # Frame pointers for perf
python3 -m spyglass build --mode memory   # jemalloc profiling support

• CPU mode: passes RUSTFLAGS="-C force-frame-pointers=yes"
• Memory mode: uses --profile release-profiling with --features jemalloc-profiling and sets JEMALLOC_SYS_WITH_MALLOC_CONF with prof:true

run

Runs Lighthouse under a profiler with a mock execution layer.

spyglass run --mode cpu -n baseline
spyglass run --mode cpu -n my-test --runs 3   # three back-to-back runs

The tool automatically:

• Starts lcli mock-el for the execution layer
• Enables the beacon HTTP API and metrics server
• Polls the beacon API to detect sync completion and epoch boundaries
• Scrapes Prometheus metrics at epoch boundaries (pre/post delta)
• Terminates cleanly on completion or Ctrl+C

analyze

Processes profiling output into flamegraphs and markdown reports.

spyglass analyze my-run --filter epoch-boundary
spyglass analyze my-run --filter all  # runs all three filters

Produces (per filter, under views/<filter>/):

• profile.collapsed — collapsed stack format
• flamegraph.svg — interactive flamegraph
• analysis.md — category breakdown + top functions

compare

Compares two profiling runs.

spyglass compare baseline optimized --filter epoch-boundary
spyglass compare baseline optimized --filter all  # compares all available views
spyglass compare --pr 6789 --filter all           # config nickname vs the pr-6789 profile

Produces a comparison.md with category-level and function-level deltas, plus a differential flamegraph. All numbers are per-run averages, so it's fine if one profile has more runs than the other. Deltas that clearly exceed the run-to-run noise are marked ✱.

If the two profiles were built from different commits, the report starts with a changed-code spotlight: spyglass finds the diff between the two commits and shows how much CPU time the changed functions cost on each side. This is the quickest way to see whether a PR did what it promised.

Useful flags:

• --pr 6789 — compare against the pr-6789 profile; the baseline defaults to the config's nickname
• --focus <regex> — add extra functions to the spotlight
• --no-spotlight — skip the spotlight section

profile

Convenience command: build + run + analyze in one step.

spyglass profile --mode cpu -n my-experiment

export

Exports profile data in formats suitable for external tools.

# Export filtered perf script text (for Firefox Profiler)
spyglass export baseline perf-script --filter epoch-boundary

# Export a filtered flamegraph SVG
spyglass export baseline flamegraph --filter epoch-boundary

# Generate an interactive HTML flame chart (epoch boundary timeline)
spyglass export baseline flamechart
spyglass export baseline flamechart --bin-size 0.25  # finer time resolution

The perf-script format can be uploaded to Firefox Profiler for interactive analysis. The flamechart format produces a self-contained HTML file with a category-coloured timeline and per-bin flamegraphs.

clean

Removes spyglass artifacts (PR checkouts and/or profiling results).

spyglass clean              # Remove PR checkouts only (default)
spyglass clean all --force  # Remove checkouts + profiles
spyglass clean profiles     # Remove only profiling results (prompts for confirmation)

run --attach

Attach to an already-running Lighthouse node instead of building and managing the runtime. Spyglass finds the process automatically by scanning /proc for a lighthouse bn process matching the network and http_port in your config.

# Attach to the local lighthouse bn (auto-detected from config)
spyglass run --mode cpu --attach -n my-profile

# Explicit PID if auto-detection can't disambiguate
spyglass run --mode cpu --attach --pid 12345 -n my-profile

After profiling completes, the lighthouse process is left running — spyglass only detaches perf.

Side-by-side profiling (stable vs unstable)

When running two simultaneous lighthouse instances on the same network (e.g. for A/B comparison), use separate configs with different http_port values to distinguish them:

# configs/stable.toml
[lighthouse]
http_port = 5052

# configs/unstable.toml
[lighthouse]
http_port = 5053

# Each command finds the right process by matching http_port
spyglass run --mode cpu --attach -c stable -n stable --runs 3
spyglass run --mode cpu --attach -c unstable -n unstable --runs 3

# Compare
spyglass analyze stable --filter all
spyglass analyze unstable --filter all
spyglass compare stable unstable --filter all

Profiling a GitHub PR

Fetch and profile a pull request directly from sigp/lighthouse:

# Profile PR #6789 (clones into checkouts/pr-6789/, builds, profiles)
spyglass profile --mode cpu --pr 6789

# Compare against your local branch
spyglass profile --mode cpu -n baseline
spyglass analyze baseline --filter all
spyglass analyze pr-6789 --filter all
spyglass compare baseline pr-6789 --filter all

The --pr flag works with any command (build, run, profile). It:

• Shallow-clones the Lighthouse repo into checkouts/pr-<number>/
• Fetches the PR ref and checks it out
• Uses that checkout for building and profiling
• Defaults the nickname to pr-<number>

Output Structure

profiles/
  <nickname-or-branch>/
    cpu/
      config.toml             # Resolved config snapshot shared by the runs (reusable with -c)
      run-1/                  # One directory per profiling run
        perf.data             # Raw perf recording
        profile.collapsed     # Full collapsed stacks (unfiltered)
        epochs.json           # Detected epoch boundaries with timestamps
        run.json              # Run metadata (config, timing, clock offset)
        metrics/
          epoch_<N>_pre.txt
          epoch_<N>_post.txt
          epoch_<N>_delta.json
        views/
          epoch_boundary/     # Filtered to epoch boundary windows
            profile.collapsed
            flamegraph.svg
            analysis.md
          mid_epoch/          # Filtered to mid-epoch periods
            ...
          steady_state/       # Filtered to steady-state periods
            ...
      run-2/
        ...
      aggregate/              # Merged views across runs + variance stats (rebuilt by analyze)
        ...
    memory/
      run-1/
        heap.*.heap
        heap_analysis.md

Configuration

Config files live in configs/. Copy the example to configs/config.toml and edit it: that's the default config, loaded whenever no -c is given.

There are three ways to vary a run on top of that, from heaviest to lightest:

1. Full config files — complete TOML files in configs/, selected with -c/--config. Bare names are resolved in configs/ (with or without the .toml extension), so -c gnosis finds configs/gnosis.toml. A path outside configs/ also works.

2. Experiments — named partial configs in configs/experiments.toml, selected with -x/--experiment. Each experiment is a table of overrides merged over the base config, so it only needs the keys that change:

   # configs/experiments.toml
   [batch-t8.lighthouse]
   extra_flags = ["--subscribe-all-subnets", "--beacon-processor-attestation-batch-trigger", "8"]

   spyglass profile --mode cpu -x batch-t8

   The run nickname defaults to the experiment name unless the experiment sets profiling.nickname itself (a CLI -n beats both). The experiments file is looked up next to whichever config -c selected.

3. CLI overrides — -n/--nickname and -o/--output-dir override the loaded config directly.

See configs/config.example.toml for all options:

[paths]
lighthouse_dir = "~/path/to/lighthouse_dir"

[lighthouse]
network = "mainnet"
checkpoint_sync_url = "https://mainnet.checkpoint.sigp.io"
extra_flags = ["--subscribe-all-subnets", "--import-all-attestations"]
http_port = 5052
metrics_port = 5054

[profiling]
perf_frequency = 1000
profile = "release"
disable_backfill = true
output_dir = "./profiles"
nickname = ""
start_slot = 16
end_slot = 15
safety_timeout = 7200

[filtering]
epoch_boundary_warmup = 6
epoch_boundary_cooldown = 6

[mock_el]
listen_address = "127.0.0.1"
listen_port = 8551

The network field is passed to both lighthouse bn and lcli mock-el. Supported values: mainnet, gnosis, holesky, sepolia, etc.

Categories

The categories.toml file defines pattern-based sample classification. Categories are checked in priority order (first match wins). See the file for the default Lighthouse categories.

Filters

Filter	Description
all	Runs all three filters below in sequence
epoch-boundary	Samples within warmup/cooldown of epoch transitions
mid-epoch	Samples NOT near epoch boundaries
steady-state	Samples after sync completes, excluding epoch boundaries

License

Apache 2.0 — see LICENSE.