Isolation probe

A TEE that hosts many apps makes a strong claim — your code ran here, the others’ code can’t tamper with yours — but the TEE attestation only covers what code is running. It does not by itself say what’s keeping co-hosted apps from each other’s secrets and files. That’s a job for the OCI runtime that brings up app containers.

dstack-webhost runs every app under a runtime the substrate selects, and exposes the choice publicly at /_api/substrate. A relying party who wants to check the claim can deploy a tiny app whose only job is to expose its own kernel-namespace view, and then compare:

A malicious substrate can lie in /_api/substrate, but it cannot forge the app’s own /proc. That’s the corroboration loop the probe demonstrates.

Live demo

The probe is deployed as a Layer-1 app on hermes-staging:

→ hermes-staging.dstack-pha-prod7.phala.network/probe/

The page fetches /_api/substrate and the app’s own /api/probe, then renders a verdict. Source: examples/isolation-probe/.

What the verdict means

uid_map is the signature. Under sysbox-runc, root inside the container (UID 0) is mapped to a non-root UID on the host — an active user-namespace remap. Plain runc shows the trivial mapping 0 0 4294967295: root inside is root outside.

sysbox-runc:   "0     231072      65536"     ← shifted, ns boundary active
runc default:  "0          0 4294967295"      ← trivial, no remap

The probe’s verdict is “consistent” when the substrate’s claim and the app’s uid_map agree on which regime is in force.

What the substrate layers on top of the runtime

The OCI runtime is one layer. dstack-webhost adds three substrate-level mechanisms that go beyond what sysbox-runc alone provides:

The shared deno runtime runs each app in its own deno run process whose permissions cover only its own files, its own /daemon-data/<name> and its own /run/broker/<name> sockets (RFC 0035). Apps there cannot derive each other’s keys or read each other’s files, env or data, but they share a network namespace and a kernel, and the boundary is Deno’s permission checker. Per-project networks and kernel-level separation require isolation: container or runtime: image.

Picking the runtime

The choice on this substrate is between three OCI runtimes, all of which can be made available on stock dstack:

For nested KVM (which would let gVisor use its faster KVM platform): not exposed in dstack TDX. The ptrace/systrap platform is what we use, with the perf cost measured below.

Recommendation

runsc (gVisor) is the recommended runtime for this substrate. Attestation is the product; the relying-party threat model genuinely values kernel-CVE resistance, which is the only thing runsc adds and sysbox-runc does not. The perf cost is ~20% throughput and ~2× p99 tail on a 1-vCPU CVM (see below) — tolerable for the small handlers and Layer-1 image apps this substrate hosts. sysbox-runc is the right fallback for apps whose workloads can’t accept that perf hit (a database under load, a latency-critical endpoint). Plain runc has no place on a multi-app TEE substrate when sysbox-runc is right there at zero cost.

This was a working assumption the substrate started under (DAEMON_CONTAINER_RUNTIME=sysbox-runc is what hermes-staging runs today); the comparison below is the data that turned it from “deferred until we measure” into a real recommendation.

Measured perf cost

A controlled comparison on the same dstack CVM (1 vCPU, 2 GB RAM), flipping DAEMON_CONTAINER_RUNTIME between the three runtimes and redeploying the probe each time. Workload: 20 concurrent clients hitting /probe/api/probe for 45s — moderately syscall-heavy (4 /proc reads + JSON serialise + HTTP serve per request).

Runtime rps p50 p90 p95 p99
runc 221.6 80.3ms 92.4ms 111.2ms 179.6ms
sysbox-runc 221.8 80.2ms 93.2ms 111.8ms 182.7ms
runsc (gVisor) 175.2 101.6ms 118.5ms 137.4ms 395.8ms

sysbox-runc is free. Indistinguishable from runc on throughput and every latency percentile. Sysbox does its hardening at container startup (user-namespace setup, proc/sys virtualisation), not per syscall — once the container is running, syscalls go straight to the host kernel like with runc. Zero per-request cost.

runsc costs about 21% throughput, 27% median latency, and 2.2× p99 tail latency. Sentry intercepts every syscall in userspace; on a 1-vCPU CVM with a syscall-heavy workload, that’s the visible price. Workloads that are mostly network-bound without much per-request kernel work will see less. Databases, file-heavy services, anything chatty with the kernel will see more.

The runtime-choice implication:

A covert channel runsc actually closes: /proc is host-wide under runc and sysbox-runc

While digging for residual cross-app channels on hermes-staging, the most striking real finding: /proc/loadavg, /proc/stat, /proc/meminfo, and /proc/uptime are all host-wide under runc and sysbox-runc. Reading them from inside any app gives back the host’s actual values — including activity contributed by sibling apps.

Side-by-side, on the same host:

                  HOST        runsc        sysbox-runc          runc
/proc/loadavg     1.89...     0.00 0.00    1.89... (host)       1.89... (host)
/proc/uptime      1001 s      0.28 s       0.05 s               1003 s (host)
/proc/stat cpu    173390      0 0 0 0      173496 (host)        173533 (host)
/proc/meminfo     2444 free   3824 (sandboxed)  2389 (host)     2379 (host)

This is a working covert channel under runc/sysbox-runc:

  1. App A busy-loops a CPU core in a deliberate pattern (e.g. spin 1s, idle 1s).
  2. App B reads /proc/loadavg once per second.
  3. B reconstructs A’s signal from the load oscillation.

Bandwidth is low (loadavg updates over seconds), but the channel is structurally there and easy to demonstrate. Sysbox doesn’t help — its FUSE-virtualised /proc covers some files but lets loadavg/stat/uptime/meminfo pass through to the host’s view.

runsc (gVisor) closes this channel by synthesising those /proc entries inside Sentry — the app only ever sees its own kernel-state-equivalent, never the host’s. Sentry IS the kernel for the workload, so there’s no host view to leak.

This is the strongest concrete reason to prefer runsc on this substrate that surfaced during the comparison work — the kernel-CVE argument was structural but abstract; this is a demonstrable cross-app information leak that flipping the runtime literally closes. (Higher-bandwidth channels — cache timing, memory pressure modulation — are harder to characterise and likely remain to some degree under any runtime when apps share physical hardware.)

Disguised leaker: a leak runtime isolation does not close

The /proc covert channel above is closed by switching runtimes. The harder threat — and the more honest one — is an app whose source review passes, whose attestation chain holds, but which leaks the very thing it claims to protect via a side channel the substrate cannot see. That’s the rug-pull case: the relying party reads the source, accepts the attestation, and is wrong.

The simplest demonstration of the class, deployed live on hermes-staging at /vault/ (under runsc + isolation: container — the strongest substrate config we have):

// vault — a credential gate. Holds a secret token; only callers who present
// the matching token via /check?t=<token> get a positive response.
function check(candidate: string, SECRET: string): boolean {
  if (candidate.length !== SECRET.length) return false;
  let ok = true;
  for (let i = 0; i < SECRET.length; i++) {
    if (candidate[i] !== SECRET[i]) {
      ok = false;
    } else {
      // jitter to harden against timing
      const end = performance.now() + 50;
      while (performance.now() < end) { /* spin */ }
    }
  }
  return ok;
}

The author’s intent reads as “iterate fully so we don’t short-circuit, add per-byte work to throw off timing analysis.” The actual effect is that response time grows linearly in the matching prefix length: each correctly-guessed byte costs an extra ~50 ms of wall clock. From the public internet, with 10 timing samples per candidate per position, an attacker recovers the secret byte-by-byte:

target: https://...phala.network/vault/  length: 8  samples/byte: 10
  pos 0: 'd'  (best=410.6ms  runner-up=378.6ms  gap=32.00ms)  so far: 'd'
  pos 1: 'e'  (best=449.1ms  runner-up=443.6ms  gap= 5.51ms)  so far: 'de'
  pos 2: 'a'  (best=511.9ms  runner-up=487.9ms  gap=24.00ms)  so far: 'dea'
  pos 3: 'd'  (best=603.3ms  runner-up=565.1ms  gap=38.16ms)  so far: 'dead'
  pos 4: 'b'  (best=614.9ms  runner-up=614.4ms  gap= 0.54ms)  so far: 'deadb'
  pos 5: 'e'  (best=717.9ms  runner-up=655.4ms  gap=62.49ms)  so far: 'deadbe'
  pos 6: 'e'  (best=717.9ms  runner-up=716.9ms  gap= 1.01ms)  so far: 'deadbee'
  pos 7: 'f'  (best=818.5ms  runner-up=774.3ms  gap=44.23ms)  so far: 'deadbeef'

recovered: 'deadbeef'  in 983.2s

Source for the project and the attack: examples/timing-leak-demo/. The substrate is doing its job — runsc mediates syscalls, per-project networks isolate the app, the daemon never sees the secret. The leak is inside the project’s own response time, public to anyone with curl.

Why this matters and what’s actually realistic

The “jitter to harden” loop in this demo is contrived — a timing-aware reviewer would catch it on the second read. The honest version is when the leak falls out of a normal cryptographic primitive without any fishy flag in the source:

The takeaway is the same in all cases: the leak is in code that looks correct to source review, the attestation chain holds, and the substrate’s runtime layer cannot see it. Closing it is application-layer work — constant-time primitives in the project’s code, audits with timing-attack literacy, ideally automated timing-property tests.

A follow-up demo with one of the above (probably textbook RSA — Hamming-weight recovery is the cleanest concrete signal to land in a docs page) is on the list. The current vault is the warmup that establishes the class.

gVisor + Docker embedded DNS

When an app runs under plain runsc, outbound DNS resolution hits a real interaction bug between gVisor’s sandbox netstack and Docker’s embedded DNS resolver at 127.0.0.11. Docker forces nameserver 127.0.0.11 into /etc/resolv.conf for any container on a user-defined bridge network (which is most of them, including the per-project tee-proj-<name>-<mode> networks the substrate creates). gVisor’s own netstack doesn’t route to that address. Result: DNS lookups return “Connection refused” even though the host’s actual resolver is reachable.

HostConfig.Dns (the compose dns: field) does not fix this — it only changes the upstream resolvers Docker’s embedded DNS forwards to, not the embedded DNS address Docker writes into /etc/resolv.conf.

The substrate’s answer is a second runtime, so no app has to choose between gVisor and DNS:

Plain runsc is unchanged: name resolution on a per-project bridge is still broken. docker_client.py sets an explicit HostConfig.Dns for runsc apps (#108), but that only changes what the embedded resolver forwards to — on a user-defined network Docker still writes 127.0.0.11, so it buys nothing there. An app that needs netstack isolation and DNS has no answer in this substrate today.

/_api/substrate reports available_runtimes read live from Docker’s GET /info, and the daemon refuses to start when DAEMON_CONTAINER_RUNTIME names a runtime Docker does not have — the substrate’s runtime claim cannot silently drift from what the host actually runs.

Open attack surface

What this stack does not address, and is worth arguing about:

A natural next demo is a sibling probe that actually tries each of these channels and reports what worked. Not built yet — the substrate-level fixes (network, volume, Deno perms) closed the easier channels first; the side-channel work is harder and lower priority.

Trying it on your own CVM

The image is already published — you can deploy it directly without rebuilding:

# Make sure the substrate is configured for sysbox-runc:
#   environment:
#     - DAEMON_CONTAINER_RUNTIME=sysbox-runc

# Deploy as an app:
curl -X POST https://<cvm>/_api/projects \
  -H "Authorization: Bearer $TEE_DAEMON_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"probe","runtime":"image",
       "image":"ghcr.io/amiller/tee-isolation-probe:v1",
       "image_port":8000}'

# Visit
open https://<cvm>/probe/

To rebuild from source (examples/isolation-probe/), see the README there.