awesome-private-inference a registry of TEE-verified inference

Methodology

The dashboard scores each (provider × model) pair across eight capability layers. This page explains exactly what each layer means and how we re-verify it.

Scorecard layers

Nonce bound
Client-supplied nonce appears in TDX report_data
TDX quote
Intel TDX quote accepted by Phala's public verifier
report_data binds key
report_data commits to signing address + nonce
GPU attested
NVIDIA NRAS returned PASS on the GPU payload
Key derives to addr
keccak(signing_public_key) == signing_address
compose_hash committed
mr_config starts with 0x01 || sha256(app_compose)
Prod OS image
vm_config.image is the production dstack OS image, not dstack-nvidia-dev. The dev image ships sshd + debug-tweaks + tools-profile; with DSTACK_AUTHORIZED_KEYS it gives the operator host-network-namespace root SSH inside the CVM (prompts in /proc//mem exfiltratable). Flips ✅ when the fleet runs dstack-nvidia-* prod.
Serving code attested
The code on the prompt-plaintext path (serve.py) and the served model are committed to a measured register. Red on Chutes: serve.py is CFSV-excluded and in no RTMR, so the operator (or Chutes) can read or exfiltrate the decrypted prompt AND silently substitute the model — neither detectable from the quote. Live proof: all -TEE models share one MRTD. chutesai/chutes#75.
Backend attested
Tri-state: ✅ gateway code self-attests + compose_hash is on-chain authorized + image digest is in our analyst-pair audit ledger. ○ chain-authorized + self-consistent but the audit ledger hasn't caught up (analyst backlog, NOT a provider fault). ❌ quote not self-consistent or compose not on the on-chain authorized set.
Attested serving forced
This hostname is in the attested tee_only_domains, so the gateway refuses to serve it from a non-TEE upstream. ❌ means the same verified workload will forward the prompt to an ordinary commercial API unless the client sets provider.aci_verified per request — live on api.redpill.ai, which shares tee.redpill.ai's quote and keyset but serves 42 models the TEE-only hosts refuse.

Independent re-verification

We do not trust the provider's own "server_verification" field when the provider exposes one (for example, Venice). We forward the TDX quote to Phala's public TDX verifier and the NVIDIA payload to NVIDIA NRAS. We then recompute key derivation and the report_data binding in-process. A green TDX cell therefore means Phala's appraisal service accepted the quote; it is not equivalent to verifying Intel's chain locally.

The current RedPill/Phala ACI gateway uses a different verifier. Its Rust client runs dcap-qvl against PCCS collateral, verifies the nonce-bound workload keyset and measured Compose, and checks the live TLS SPKI. That ACI surface is documented in the provider pages but is not yet a daily dashboard row. Its separately reproduced deployment verdict is Stage 0; the 5/6 ACI protocol result is not a Stage score.

Known upstream TODOs

Phala's private-ai-verifier decodes NRAS and Intel Trust Authority JWTs with verify_signature=False. Provider adapters inherit that limitation when they rely on those decoded claims. The ACI client's native DCAP verification of the gateway quote is a separate path. See devproof-audits-guide/LEARNINGS.md.

Why "Stage 0" on every provider

Even when every layer in the matrix shows ✅, the provider is Stage 0 in the ERC-733 / devproof sense for at least one of:

How Backend attested is decided

For providers with a gateway→backend hop, we cross-reference three signals before marking the column. The cell is tri-state, and the middle state is deliberate: we want to distinguish "provider shipped code we haven't audited yet" from "provider is doing something wrong."

Reproduce

git clone https://github.com/amiller/awesome-private-inference
cd awesome-private-inference
uv venv && source .venv/bin/activate
uv pip install -e .

git clone https://github.com/nearai/nearai-cloud-verifier _nearai-verifier
export NEARAI_VERIFIER_PATH="$(pwd)/_nearai-verifier/py"

export NEAR_API_KEY=...
export VENICE_API_KEY=...

python -m probes.collect
python -m probes.render

# Current RedPill/Phala ACI surface, verified separately from this Python matrix
git clone https://github.com/Dstack-TEE/private-ai-gateway.git
cd private-ai-gateway
cargo run --bin aci -- verify https://tee.redpill.ai
cargo run --bin aci -- verify https://tee.redpill.ai --require-production-os
cargo run --bin aci -- sessions https://tee.redpill.ai

The production-OS option is an RTMR3 appraisal and currently rejects the live gateway. Use it with a dstack verifier that reconstructs MRTD and RTMR0-2 from the same quote, event log, and VM configuration.