Skip to content

tunneld: Reach remote and federated tunnelds over the /connect bridge - #1896

Merged
doronz88 merged 7 commits into
masterfrom
feature/tunneld-remote-bridge
Aug 31, 2026
Merged

tunneld: Reach remote and federated tunnelds over the /connect bridge#1896
doronz88 merged 7 commits into
masterfrom
feature/tunneld-remote-bridge

Conversation

@doronz88

Copy link
Copy Markdown
Owner

Builds on #1893: /connect exposed the tunnel over HTTP, but nothing in pymobiledevice3 consumed
it, so a remote tunneld was only reachable by writing your own websocket client. This makes the
client speak it, and makes a tunneld able to front other tunnelds over the same bridge.

What

Reach a remote tunneld (--tunnel UDID@HOST[:PORT]). TunneldConnectDialer is an
asyncio.open_connection-compatible dialer plugged into the seam RemoteServiceDiscoveryService
already exposes for the userspace tunnel, so there is no new call site: every device connection an
RSD opens is bridged through /connect. The caller is handed one end of a socket.socketpair
whose other end is pumped against the websocket, so consumers that reach for
writer.get_extra_info('socket') (ServiceConnection) keep working. Bridging is automatic when
the tunneld host is non-loopback, and bridge=True (the @ form on the CLI) forces it for
addresses that only look local, such as an SSH port-forward.

Front several hosts with one tunneld (--upstream, repeatable). Federation previously only
merged an upstream's listing, and the addresses in that listing are ULAs on the upstream host's
utun — reachable only from a client that already had L3 connectivity to that host. So it
presupposed a VPN or hand-rolled routes, and a device discovered through an aggregator was a dead
end. A /connect for a device the instance does not serve itself is now relayed to the upstream
that owns it:

# every host with devices attached
sudo pymobiledevice3 remote tunneld --host 0.0.0.0

# the one host clients can reach (monitors off: it federates rather than competing for devices)
sudo pymobiledevice3 remote tunneld --host 0.0.0.0 \
    --no-usb --no-wifi --no-usbmux --no-mobdev2 \
    --upstream http://lab-1:49151 --upstream http://lab-2:49151

# from anywhere
pymobiledevice3 developer dvt ls / --tunnel 'UDID@front:49151'

GET / entries now carry an originnull when served directly, else the URL of the hop that
reported them — so a client that can reach that upstream may skip the relay. Upstream close codes
are passed through, so a 4404/4502 names the tunneld that actually refused.

Remove --uds. It only ever moved the HTTP control channel onto a unix socket while the data
plane (the RSD dials, and now the bridge) always relied on TCP, so it never delivered a pure-unix
path. TunneldAddress narrows to (host, port); a --tunnel value with a non-numeric :suffix
now fails with a message naming the removal instead of being read as a socket path.

Bugs found and fixed along the way

  • Federation cycles dropped the listings they were meant to merge. Tunnelds registering each
    other recursed on every GET /: the nesting spawned a growing tree of in-flight requests against
    the bounded to_thread executor, and the innermost fetches lost the race against their parents'
    2s timeout. With two mutually registered tunnelds, each one's listing was missing the other's
    device
    . Requests now carry a hop budget (x-tunneld-hops-remaining, 4), and a tunnel reported
    through several paths is listed once.
  • /connect addressed a device, not a tunnel. A device can have more than one tunnel — a front
    tunneld that also monitors the same host holds its own alongside the federated one. Keying on
    the UDID alone paired whichever tunnel was found locally with the port the caller asked for,
    dialing an endpoint that exists nowhere (4502 failed to connect to [fdc9:…]:63727). /connect
    now takes an optional ?address=; older tunnelds ignore it and keep their UDID-only behaviour.
  • Upstream URLs failed silently. A schemeless lab-1:49151 parses as scheme lab-1 with no
    host, so the listing fetch threw InvalidSchema into a debug log and the relay refused —
    registering one did nothing at all. https:// fetched over TLS but relayed in plaintext to port
    80. Registration now normalizes HOST, HOST:PORT and http://HOST[:PORT] to one canonical
    form and rejects what neither path could act on, as a usage error on --upstream and a 400 from
    POST /upstream.
  • Abandoned bridges spewed teardown errors. Ctrl+C on a live stream (syslog live) left the
    pump coroutines to GC, which closes them with GeneratorExit, where the awaits in their finally
    blocks raised coroutine ignored GeneratorExit. Teardown is synchronous now.

Verification

Against a real iPhone (iOS 26.6.1) over USB, not only fakes:

  • dvt ls / and dvt device-information through the bridge (--tunnel UDID@127.0.0.1, and the
    empty-UDID @host form), with zero tracebacks in the tunneld log.
  • The same commands through an aggregator holding no tunnels of its own while fronting a
    separate tunneld process — a topology that is self-proving, since it has no other way to reach
    the device.
  • The ?address= fix in the failing topology: an aggregator holding both its own tunnel and a
    federated one to the same device now serves dvt ls / over either, where both previously failed.

Automated: 704 tests pass (pytest -m "not device"), including new suites for the bridge
(tests/test_tunneld_remote_bridge.py) and federation (tests/test_tunneld_federation.py) that
spin up real tunneld servers over uvicorn/wsproto against a TCP echo server. The cycle and
?address= regressions were both confirmed to fail before their fixes. pyright --venvpath .
(1.1.411) reports 0 errors; ruff check / ruff format --check (0.15.4) and markdownlint are clean.

Notes

  • /connect remains unauthenticated like the rest of the tunneld API, and an aggregator relays
    into other hosts' devices, which widens that exposure. The auth work offered in tunneld: Add /connect websocket endpoint exposing the tunnel over HTTP #1893 should
    land before this topology is advertised broadly; the docs carry the warning meanwhile.
  • Relaying costs an extra hop through the front instance's event loop — negligible for control
    traffic, measurable on bulk transfers such as DDI mounts.
  • Unrelated observation from tunneld: Add /connect websocket endpoint exposing the tunnel over HTTP #1893's test suite: on macOS, connecting to a socket held bound but
    not listening
    gets silence rather than an RST, so test_connect_tcp_connection_refused burns the
    full ~7.8s connect timeout on every run there.

Comment thread pymobiledevice3/tunneld/server.py Dismissed
The feature was stale by design: --uds only ever moved the HTTP control
channel onto a unix socket, while the data plane - the RSD dials to the
reported tunnel-address:tunnel-port - always relied on TCP, so it never
delivered a pure-unix path.

- remote tunneld loses --uds; TunneldRunner binds TCP only
- TunneldAddress narrows to (host, port); the UDS client transport
  (_UnixHTTPConnection) is gone
- a --tunnel value with a non-numeric :suffix now fails with a clear
  BadParameter naming the removal, instead of being treated as a socket
  path
pymobiledevice3 now consumes the WS /connect endpoint itself, so every
command and library consumer works against a tunneld whose tunnel
interface is unreachable from the client - tunneld in a different docker
network stack, on another host, or behind an SSH port-forward.

- TunneldConnectDialer: an asyncio.open_connection-compatible dialer
  (the seam RemoteServiceDiscoveryService already exposes for the
  userspace tunnel) that bridges each device-bound connection through
  /connect. The caller gets one end of a socket.socketpair whose other
  end is pumped against the websocket - a real socket, so
  ServiceConnection's get_extra_info('socket') contract holds. Each
  pump ends when either leg closes, so closing the RSD tears down
  every bridge; nothing needs explicit teardown.
- get_tunneld_devices()/get_tunneld_device_by_udid() take
  bridge=None|bool: None bridges automatically when the tunneld host is
  non-loopback; True forces it for addresses that only look local
  (SSH port-forwards).
- --tunnel accepts UDID@HOST[:PORT] (IPv6 in brackets), which forces
  the bridge; a tunneld that predates /connect fails with a clear
  upgrade message.

Verified end-to-end against a real tunneld server (uvicorn/wsproto)
bridging into a local TCP echo server, including through
ServiceConnection.create_using_tcp.
Tunnelds that register each other (A -> B -> A) recursed on every GET /:
the nesting spawned a growing tree of in-flight requests against the
bounded to_thread executor, and the innermost fetches lost the race
against their parents' 2s timeout - so a cycle dropped the very listings
it was meant to merge. With two mutually registered tunnelds, each one's
listing was missing the other's device.

Federated requests now carry x-tunneld-hops-remaining (default 4),
decremented per hop and not forwarded at zero, and entries reported
through several paths are listed once.

Verified with two real federated tunneld servers: the peer's device is
now present (it was missing before) and each tunnel appears once.
Federating another tunneld only merged its listing, and the addresses in
that listing are ULAs on the upstream host's utun - reachable only from a
client that already had L3 connectivity to that host. So federation
presupposed a VPN or hand-rolled routes, which is why a device discovered
through an aggregator was a dead end.

The /connect bridge removes that prerequisite. A /connect for a device
none of this instance's own tunnels serve is now relayed to the upstream
that owns it, so a client needs a route to one tunneld and nothing else:

    sudo pymobiledevice3 remote tunneld --host 0.0.0.0 \
        --upstream http://lab-1:49151 --upstream http://lab-2:49151
    pymobiledevice3 developer dvt ls / --tunnel 'UDID@front:49151'

- new --upstream (repeatable) seeds the set POST /upstream manages
- GET / entries carry an origin: null when served directly, else the URL
  of the hop that reported them, so a client that can reach that upstream
  may skip the relay
- the upstream's own close code is passed through, so 4404/4502 name the
  tunneld that actually refused
- the /connect client is now shared (tunneld/ws_bridge.py) between the
  dialer and this relay rather than hand-rolled twice

Verified against a real device: an aggregator holding no tunnels of its
own fronted a separate tunneld process and served dvt ls / and
device-information over the two-hop path, with no tracebacks on either
instance.
Federation reads an upstream address twice - requests fetches the listing
from the URL, while the /connect relay dials the host and port parsed out
of it - and the two disagreed on everything except the exact spelling
'http://HOST:PORT':

- a schemeless 'lab-1:49151' (the spelling --tunnel UDID@HOST:PORT
  invites) parses as scheme 'lab-1' with no host: the listing fetch threw
  InvalidSchema into a debug log and the relay refused, so registering
  one did nothing at all, silently
- an https:// upstream fetched its listing over TLS but was relayed to in
  plaintext, on port 80 when no port was given
- a URL carrying a path fetched that path but relayed to the bare host

Registration now runs both spellings through one normalizer, which fills
in tunneld's default port, brackets IPv6 literals, and rejects what
neither path could act on (non-http scheme, no host, a path or query, a
bad port). --upstream reports the failure as a usage error and
POST /upstream as a 400, instead of accepting an address that quietly
never works. HOST, HOST:PORT and http://HOST[:PORT] are all accepted and
stored canonically, so GET /upstream and DELETE /upstream agree.
A device can have more than one tunnel: when a tunneld federates another
that monitors the same host, its listing carries both its own tunnel and
the federated one for that UDID. /connect keyed on the UDID alone, so it
paired whichever tunnel it found locally with the port the caller asked
for - dialing one tunnel's address at another's port, an endpoint that
exists nowhere:

    4502 failed to connect to [fdc9:ec7b:4fdf::1]:63727: [Errno 61]
         Connect call failed

/connect now takes an optional ?address=, the tunnel address the caller
means. A local tunnel is used only when it matches; otherwise the request
falls through to the upstream that serves that exact tunnel, so holding
our own tunnel to a device no longer hides another instance's. The dialer
names the tunnel it is bridging for, and the relay passes it upstream.
Older tunnelds ignore the parameter and keep their UDID-only behaviour.

Docs: a front tunneld on a host that also has devices attached should run
with the monitors off, or it federates a device it is also serving and
lists it twice.

Verified against a real device in that exact topology: an aggregator
holding its own tunnel plus a federated one now serves dvt ls / over
either, where both previously failed.
@doronz88
doronz88 force-pushed the feature/tunneld-remote-bridge branch from 8087cfd to 44ba610 Compare August 31, 2026 07:35
The test held a socket bound but not listening and connected to its port,
expecting an RST. macOS does not send one - it drops the SYN silently -
so tunneld burned its whole CONNECT_TCP_TIMEOUT before answering 4502,
and on the slower macOS runners the test client's own recv timeout fired
first: 'socket.timeout: timed out' on py3.9/macos-latest.

Connect to a released port instead, which does get an RST. The file drops
from 9.3s to 1.5s as a side effect. A port stolen in the gap makes the
assertion fail loudly rather than pass silently.
@doronz88
doronz88 merged commit ec4ac06 into master Aug 31, 2026
27 checks passed
@doronz88
doronz88 deleted the feature/tunneld-remote-bridge branch August 31, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants