Compare commits

...

34 Commits

Author SHA1 Message Date
redshift
b81c5add6a test: satisfy mypy usage parser assertions 2026-07-10 23:40:37 +08:00
redshift
02109616b9 fix(ehbp): compare resolved upstream model identities 2026-07-10 15:16:21 +08:00
redshift
b30346bcb6 chore: untrack stray local files & gitignore
Remove accidentally-committed local-only files:
- .wallet/wallet.sqlite3-shm / .wal (runtime DB sidecar files)
- AGENTS.md, TEST_SUITE_OVERVIEW.md (local agent context artifacts)

Add .wallet/, AGENTS.md, TEST_SUITE_OVERVIEW.md to .gitignore.
2026-07-10 10:17:22 +05:30
redshift
956a1ac3e1 fix(ehbp): harden model comparison against casing & date-versioned aliases
- Case-insensitive comparison of served model vs forwarded_model_id
  (matches get_model_instance's lowercasing semantics)
- Suppress spurious mismatch when get_model_instance resolves back to
  the same model (e.g. date-versioned glm-5-2-20260415 -> glm-5-2)
- Preserve unknown-model warning only when lookup returns None
- Tests: case-insensitive match + date-versioned alias resolution
2026-07-10 09:58:35 +05:30
redshift
4287f038cf feat(ehbp): extract & use model name from Tinfoil usage metrics header
Tinfoil PR #385 added model=<name> to the X-Tinfoil-Usage-Metrics
header/trailer.  This commit uses that field for accurate billing.

Changes in routstr/upstream/ehbp.py:

- parse_tinfoil_usage_metrics(): extract the model= field as a string
  alongside the existing token counts (previously silently discarded
  because int() failed on it).

- _build_cost_info(): accept optional actual_model parameter propagated
  through to callers when a real mismatch is detected.

- _compute_ehbp_actual_cost(): compare the served model against
  model_obj.forwarded_model_id (the expected upstream ID) rather than
  model_obj.id (the client-facing alias).  This prevents spurious
  mismatches when a node runner maps e.g. tinfoil-glm-5-2 -> glm-5-2
  and the header correctly reports glm-5-2.  On a genuine mismatch
  (failover to a different upstream model), look up the actual model's
  pricing via get_model_instance() (forwarded_model_id values are
  registered as routable aliases in the global model map).

- forward_ehbp_request() / forward_ehbp_x_cashu_request(): use the
  actual served model for payment finalization and logging when a
  mismatch is detected.

Tests: 6 new scenarios (alias match, real mismatch with alias, unknown
model fallback, old-format compat, cache token details + model), plus
forwarded_model_id set on all existing mock model objects to keep them
passing.  All 49 Tinfoil/EHBP unit tests pass.
2026-07-10 09:57:10 +05:30
redshift
82fd2c08a7 Merge branch 'main' of https://github.com/Routstr/routstr-core into ehbp-proxy-refactor 2026-07-10 12:12:51 +08:00
redshift
0f7d3d2f86 fix(wallet): return 425 when refund is pending, not 404
The X-Cashu refund endpoint raised 404 "Refund not found" when the
"in" transaction existed with a request_id but the "out" (refund)
transaction had not been written yet. This is a timing race: the
endpoint is polled while the upstream request is still in flight,
before send_refund() has minted and stored the refund token.

The 404 was indistinguishable from a genuinely-missing refund, so
clients had no signal that retrying would succeed — leading to
stranded refunds when clients gave up.

Replace the third 404 branch with 425 Too Early + Retry-After: 2.
The two earlier 404 branches (no "in" row, no request_id) remain 404
since those genuinely mean no refund will ever exist.

Also adds debug logging on all three not-found/pending branches so
the race is no longer invisible to operators (middleware does not log
request headers).

Adds unit tests for the new 425 pending path and the no-request_id
404 path.

Refs: refund-race-condition
2026-07-07 22:00:21 +08:00
redshift
af2abc3a1c Rename TinfoilUpstreamProvider.from_db_row to _build_from_row
Aligns Tinfoil with the base class hook pattern: subclasses override
_build_from_row so the base from_db_row wrapper stamps db_id onto
the instance. Previously Tinfoil overrode from_db_row directly,
bypassing the identity-stamping wrapper.
2026-07-07 21:15:34 +08:00
redshift
bc6d0af6d6 Merge branch 'main' of github.com-red:Routstr/routstr-core into ehbp-proxy-refactor 2026-07-07 13:47:06 +05:30
redshift
65ac1b0dcd remove test for .well-known/ bypass (path no longer routed)
The .well-known/ path was removed from Tinfoil attestation routing
since Tinfoil doesn't use it. The test that asserted GET /.well-known/
bypasses model lookup is no longer valid.
2026-07-02 20:15:00 +05:30
redshift
ab5596ed70 remove .well-known/ from Tinfoil attestation routing
Tinfoil's proxy server guide only requires /attestation (GET) and
/v1/chat/completions + /v1/responses (POST). The .well-known/ path was
never requested by Tinfoil and was incorrectly added to:
- _API_PATH_PREFIXES (prefix gate)
- the unauthenticated GET bypass branch
- the Tinfoil integration docs

Remove it from all three.
2026-07-02 20:09:58 +05:30
redshift
55622cb956 Merge pull request #580 from Routstr/refactor-pr575-review
Reviewing the changes
2026-07-02 14:37:16 +00:00
9qeklajc
7328a5ac45 Address EHBP review comments 2026-07-01 22:41:14 +02:00
9qeklajc
07d39c2a7b simplify 2026-06-30 23:50:25 +02:00
redshift
bd4f4e8207 fix: expose Ehbp-Response-Nonce and Ehbp-Encapsulated-Key in CORS
Browser clients need these EHBP protocol headers visible to JavaScript
so the Tinfoil SDK can detect and decrypt encrypted responses. Without
them, CORS hides the headers, the SDK treats the response as a plaintext
proxy error, and users see 'The provider did not respond to this request.'

Node.js scripts are unaffected (no CORS enforcement).
See ../routstr-chat/TINFOIL_CORS_ISSUE.md for full root-cause analysis.
2026-06-29 16:33:25 +08:00
redshift
46b1f8f72d chore: remove secp256k1 git source pin 2026-06-29 13:51:31 +08:00
redshift
1c0570cc0e Merge branch 'main' of github.com-red:Routstr/routstr-core into ehbp-proxy-refactor 2026-06-29 13:33:56 +08:00
redshift
a8f60643e0 feat: return per-request cost headers on EHBP/Tinfoil responses
EHBP response bodies are opaque encrypted blobs, so cost cannot be injected
into JSON like the normal proxy flow. Instead, surface cost as response headers:

  X-Routstr-Cost-Msats       — total msats charged (bearer + x-cashu)
  X-Routstr-Cost-Usd         — USD equivalent (bearer only)
  X-Routstr-Input-Cost-Msats — msats attributed to input tokens
  X-Routstr-Output-Cost-Msats— msats attributed to output tokens

- Refactor _compute_ehbp_actual_cost from int→dict to carry full cost breakdown
- Add _build_cost_info() and _inject_cost_response_headers() helpers
- Capture adjust_payment_for_tokens return in bearer path (was discarded)
- Update CORS expose_headers, docs, and unit tests
2026-06-23 20:42:01 +08:00
redshift
e4753b5864 fix: stream buffered EHBP body instead of returning fixed-length Response
The Tinfoil SDK expects chunked transfer encoding for streaming
responses. Returning Response with content=body caused the stream to
terminate abruptly instead of ending gracefully. Use StreamingResponse
with a generator that yields the buffered body so the client gets
proper chunked transfer encoding while still benefiting from trailer
capture and exact billing.
2026-06-23 15:00:41 +08:00
redshift
66d7bd87b5 fix: capture HTTP trailers for EHBP streaming usage metrics
Tinfoil returns X-Tinfoil-Usage-Metrics as an HTTP trailer on streaming
responses, but httpx/httpcore silently discard trailers during chunked
transfer decoding. This caused all streaming EHBP requests to fall back
to max-cost billing instead of charging actual token usage.

- Add routstr/upstream/tinfoil_trailer.py: h11-based HTTP client that
  preserves trailers via the EndOfMessage event (httpx discards them)
- Rewrite forward_ehbp_request and forward_ehbp_x_cashu_request to use
  forward_with_trailer instead of httpx
- Add _extract_usage_from_response() to check both response headers
  (non-streaming) and trailers (streaming) for usage metrics
- Buffer EHBP response bodies (acceptable since they are opaque
  encrypted blobs the client decrypts regardless)
- Both bearer and X-Cashu paths now bill based on actual token usage
  for both streaming and non-streaming requests
2026-06-23 14:44:57 +08:00
redshift
4c48df8aa8 fix: remove unused Field import after mypy fix 2026-06-22 21:08:57 +08:00
redshift
abc1ea5b35 fix: resolve all mypy errors (Field defaults, missing return types, ASGITransport arg-type) 2026-06-22 21:04:51 +08:00
redshift
52dd011cd8 fix: add TYPE_CHECKING import for EHBPForwardingTarget (ruff F821) 2026-06-22 20:55:05 +08:00
redshift
94a3215894 docs: fix forward_get_request docstring for Tinfoil
The docstring claimed X-Tinfoil-Enclave-Url is honored for GET requests,
but the implementation delegates to the base class which builds URLs from
self.base_url. X-Tinfoil-Enclave-Url is an EHBP-only header for encrypted
POST requests and is not used for unencrypted GETs.
2026-06-22 16:37:13 +08:00
redshift
5dd3cbbc22 fix: route Tinfoil attestation directly 2026-06-22 13:40:32 +08:00
redshift
55fc25b4de fix: preserve provider EHBP target headers 2026-06-22 12:08:03 +08:00
redshift
36ed3ec9f0 fix: validate Tinfoil EHBP enclave URL overrides 2026-06-22 12:04:06 +08:00
redshift
2e350e082b feat: add Tinfoil direct blind-upstream integration
Add TinfoilUpstreamProvider that uses inference.tinfoil.sh as a direct
EHBP upstream. Routstr acts as a blind relay: it forwards the opaque
encrypted body to the Tinfoil enclave without ever seeing plaintext,
and bills from the X-Tinfoil-Usage-Metrics response header.

- New routstr/upstream/tinfoil.py: fetches models from public
  GET /v1/models, parses Tinfoil pricing into standard Model/Pricing
  schema, supports_ehbp=True, proxies /attestation to atc.tinfoil.sh
- Updated routstr/upstream/ehbp.py:
  - parse_tinfoil_usage_metrics() parses prompt=N,completion=N header
  - _resolve_ehbp_target_url() honors X-Tinfoil-Enclave-Url from SDK
  - _strip_proxy_headers() removes proxy-only headers before forwarding
  - _compute_ehbp_actual_cost() converts usage to msats via calculate_cost
  - forward_ehbp_request() finalizes with exact token cost when usage
    header is present (non-streaming), falls back to max-cost otherwise
  - forward_ehbp_x_cashu_request() computes refund from actual cost
    when usage is available
- Updated routstr/proxy.py: forward /attestation and /.well-known/ paths
  without model/cost/auth lookups
- Updated routstr/upstream/__init__.py and helpers.py: register and
  auto-seed Tinfoil provider from TINFOIL_API_KEY env var
- Updated .env.example with TINFOIL_API_KEY
- 22 new unit tests in tests/unit/test_tinfoil_integration.py
- Updated docs/tinfoil-direct-integration.md and docs/ehbp-proxy-support.md
  with implementation status and billing behavior table
2026-06-21 13:16:01 +08:00
redshift
24b90af6e6 refactor: move EHBP logic to dedicated module with explicit opt-in
- Add supports_ehbp + get_ehbp_forwarding_target hooks to BaseUpstreamProvider,
  keeping base.py minimal (no large forwarding methods)
- Create routstr/upstream/ehbp.py with forward_ehbp_request and
  forward_ehbp_x_cashu_request helpers, plus max-cost finalization
- PPQAIUpstreamProvider: set supports_ehbp = True, implement target to
  /private/v1/... with X-Private-Model header
- proxy.py: filter to EHBP-capable providers, route to ehbp helpers
- Fix bearer EHBP payment: reserve upfront, finalize max cost on success
- Fix X-Cashu EHBP: refund full token on failure, refund excess on success
- Update docs/ehbp-proxy-support.md to reflect new architecture
2026-06-20 17:18:53 +08:00
redshift
b9bf0ea23b Merge branch 'main' of https://github.com/Routstr/routstr-core 2026-06-20 11:07:05 +08:00
redshift
11fc826bfb Merge branch 'main' of https://github.com/Routstr/routstr-core 2026-06-15 15:43:55 +08:00
redshift
757d4400af Merge branch 'main' of https://github.com/Routstr/routstr-core 2026-06-12 10:18:22 +08:00
redshift
766d59d666 Merge branch 'main' of https://github.com/Routstr/routstr-core 2026-06-04 16:08:28 +08:00
redshift
98ddd37853 2 weeks 2026-06-04 16:07:23 +08:00
19 changed files with 3558 additions and 20 deletions

View File

@@ -2,6 +2,9 @@
UPSTREAM_BASE_URL=https://api.openai.com/v1
UPSTREAM_API_KEY=your-upstream-api-key
# Tinfoil (confidential inference enclaves, EHBP)
# TINFOIL_API_KEY=your-tinfoil-api-key
# ADMIN_PASSWORD=secure-admin-password
# Database

4
.gitignore vendored
View File

@@ -16,6 +16,9 @@ dist/
*.db-shm
*.db-wal
.*wallet.sqlite3
.wallet/
AGENTS.md
TEST_SUITE_OVERVIEW.md
*models.json
.cashu
.relay
@@ -38,3 +41,4 @@ proof_backups
*.todo
ui_out
.worktrees

158
docs/ehbp-proxy-support.md Normal file
View File

@@ -0,0 +1,158 @@
# EHBP Proxy Support for Tinfoil Models
## Problem
The SDK's `SecureClient.fetch` encrypts request bodies with HPKE (EHBP protocol)
and sends them to the Routstr provider. The Routstr proxy had no EHBP handling:
1. It tried to `json.loads()` the binary HPKE-sealed body → failed with a 400
2. The upstream PPQ.AI public endpoint (`/v1/chat/completions`) doesn't speak
EHBP, so the response had no `Ehbp-Response-Nonce` header
3. `SecureClient` threw `Missing Ehbp-Response-Nonce header` because it expects
every response from an EHBP-configured `baseURL` to carry that header
## Root cause
PPQ.AI exposes EHBP-aware inference at `/private/`, separate from the public
`/v1/` endpoint. The Routstr proxy was forwarding to `/v1/` (the public
endpoint) instead of `/private/` (the enclave endpoint). The public endpoint
can't decrypt the body, returns a normal HTTP response, and the SDK can't
decrypt it because there's no nonce header.
The PPQ private-mode proxy (`ppq-private-mode-proxy/lib/proxy.ts`) shows the
correct pattern: `SecureClient` talks to `api.ppq.ai/private/v1/chat/completions`,
which decrypts inside the attested enclave and returns an EHBP-encrypted
response with the `Ehbp-Response-Nonce` header.
## What was changed
### `routstr/proxy.py`
Detects EHBP requests by checking for the `Ehbp-Encapsulated-Key` header (set
by the EHBP transport on every encrypted request). For EHBP requests:
- Skips JSON body parsing (the body is binary ciphertext, not JSON)
- Reads the model ID from the `X-Routstr-Model` header (set by the SDK) instead
of from `body.model`
- Routes through new `forward_ehbp_request` (bearer auth) and
`forward_ehbp_x_cashu_request` (x-cashu auth) methods
- Skips reactive 400 param correction (can't parse encrypted response body)
- Still charges the user via `pay_for_request` (uses `max_cost_for_model` from
the model registry, not the body)
### `routstr/upstream/base.py`
Keeps EHBP as an explicit opt-in provider capability instead of making every
upstream provider appear EHBP-capable:
- `supports_ehbp = False` by default
- `get_ehbp_forwarding_target(path, model_obj)` raises `NotImplementedError`
unless a provider opts in and returns a provider-specific EHBP target
The actual EHBP forwarding logic does **not** live in `base.py`.
### `routstr/upstream/ehbp.py`
Contains the shared opaque EHBP transport helpers:
- `EHBPForwardingTarget` — provider-specific target URL plus extra headers
- `forward_ehbp_request()` — forwards the raw encrypted body to an EHBP-capable
provider, streams the encrypted response back untouched, and finalizes bearer
billing at max cost because usage is encrypted
- `forward_ehbp_x_cashu_request()` — redeems the Cashu token, forwards raw,
refunds the full token on upstream failure, and refunds any value above
`max_cost_for_model` on success
### `routstr/upstream/ppqai.py`
- Sets `supports_ehbp = True`.
- Implements `get_ehbp_forwarding_target()` to forward to
`https://api.ppq.ai/private/v1/...` — the PPQ.AI enclave endpoint that
understands EHBP and returns the `Ehbp-Response-Nonce` header.
- Adds `X-Private-Model` with the model's `forwarded_model_id` (e.g.
`private/kimi-k2-6`). PPQ.AI's billing layer needs this since it can't
decrypt the body.
## Why it's done this way
The proxy is a **blind relay** for EHBP requests. It cannot decrypt the body
(only the attested enclave can), so it must:
1. Get the model ID from a header, not the body
2. Forward the raw bytes without parsing or transformation
3. Stream the response back without SSE/cost parsing
4. Pass through EHBP protocol headers (`Ehbp-Encapsulated-Key` on request,
`Ehbp-Response-Nonce` on response)
Cost tracking happens at the proxy level using `max_cost_for_model` from the
model registry. Because EHBP responses are encrypted, Routstr cannot reconcile
against token usage. Bearer requests reserve and then finalize max-cost billing;
X-Cashu requests redeem the token and refund any amount above max cost.
## End-to-end flow
```
SDK Routstr Proxy PPQ.AI /private/
│ │ │
│── X-Routstr-Model: tinfoil-kimi-k2-6 ─│ │
│── Ehbp-Encapsulated-Key: <hex> ───────│ │
│── Authorization: Bearer <cashu> ──────│ │
│── body = HPKE-encrypted(kimi-k2-6) ───│ │
│ │ │
│ detects Ehbp-Encapsulated-Key │
│ reads model from X-Routstr-Model │
│ does billing/routing │
│ │ │
│ adds X-Private-Model: private/kimi-k2-6
│ forwards raw body to /private/v1/... │
│ │──────────────────────────────▶│
│ │ enclave decrypts
│ │ runs inference
│ │◀── Ehbp-Response-Nonce ──────│
│ │◀── encrypted response ────────│
│ │ │
│ streams response back untouched │
│◀── encrypted response ────────────────│ │
│ │ │
SecureClient reads nonce, decrypts │ │
SDK SSE processing sees plaintext │ │
```
## Model ID mapping
Three parties see three different model IDs:
| Party | Header/Body | Value | Source |
|---|---|---|---|
| Routstr proxy | `X-Routstr-Model` header | `tinfoil-kimi-k2-6` | SDK sends full caller-facing id |
| PPQ.AI billing | `X-Private-Model` header | `private/kimi-k2-6` | Proxy sends `forwarded_model_id` |
| Tinfoil enclave | `body.model` (encrypted) | `kimi-k2-6` | SDK strips `tinfoil-` prefix before encryption |
## Implementation status
A dedicated `TinfoilUpstreamProvider` (`routstr/upstream/tinfoil.py`) now
implements the direct blind-upstream pattern described above. The shared EHBP
helpers in `routstr/upstream/ehbp.py` were extended to:
- Request usage metrics via `X-Tinfoil-Request-Usage-Metrics: true`.
- Parse `X-Tinfoil-Usage-Metrics` from the response header (non-streaming).
- Override the forwarding URL with `X-Tinfoil-Enclave-Url` when the SDK sends it.
- Finalize bearer billing with actual token cost via `adjust_payment_for_tokens`.
- Compute X-Cashu refunds from actual cost instead of max cost.
See `docs/tinfoil-direct-integration.md` for the full implementation notes.
## Not yet tested
These changes were written without integration testing due to the complexity
of the full stack (SDK + proxy + PPQ.AI enclave + Cashu mint). Needs end-to-end
verification with a real `tinfoil-*` model request.
Important assumptions to verify:
- PPQ.AI accepts `/private/v1/...` with `X-Private-Model`.
- PPQ.AI enforces consistency between `X-Private-Model` and the encrypted
`body.model`, otherwise a malicious client could understate
`X-Routstr-Model` for billing.
- SDK behavior on non-2xx proxy-generated errors that do not carry
`Ehbp-Response-Nonce`.

View File

@@ -0,0 +1,493 @@
# Tinfoil / PPQ Private-Mode Integration Notes
This document summarizes the current options for integrating Tinfoil/PPQ private models with Routstr, based on the EHBP work in this branch and local testing against `ppq-private-mode-proxy`.
## Background
PPQ private models run behind a Tinfoil/EHBP flow:
- Request bodies are HPKE-encrypted by a Tinfoil client.
- The PPQ `/private/` endpoint routes ciphertext to the attested enclave.
- The enclave decrypts, runs inference, and returns an encrypted response.
- The caller's Tinfoil client decrypts the response locally.
The PPQ private-mode proxy (`~/projects/ppq-private-mode-proxy`) uses this pattern with the JavaScript `tinfoil` SDK:
```ts
import { SecureClient } from "tinfoil";
const apiBase = "https://api.ppq.ai";
const client = new SecureClient({
baseURL: `${apiBase}/private/`,
attestationBundleURL: `${apiBase}/private`,
transport: "ehbp",
});
await client.ready();
const response = await client.fetch(`${apiBase}/private/v1/chat/completions`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.PPQ_API_KEY}`,
"X-Private-Model": "private/gpt-oss-120b",
"x-query-source": "api",
},
body: JSON.stringify({
model: "gpt-oss-120b", // enclave-internal model id
messages: [{ role: "user", content: "Hello" }],
}),
});
const json = await response.json();
console.log(json.usage);
```
`X-Private-Model` carries the PPQ-facing private model id, while the encrypted JSON body uses the enclave-internal model id without the `private/` prefix.
## PPQ private model pricing
PPQ exposes private model pricing through:
```text
GET https://api.ppq.ai/v1/models?type=all
```
Filter models whose IDs start with `private/`.
Example private pricing observed:
| Model | Input USD / 1M tokens | Output USD / 1M tokens |
|---|---:|---:|
| `private/gpt-oss-120b` | `0.79125` | `1.31875` |
| `private/llama3-3-70b` | `1.84625` | `2.90125` |
| `private/qwen3-vl-30b` | `1.31875` | `4.22` |
| `private/glm-5-2` | `1.5825` | `5.53875` |
| `private/gemma4-31b` | `0.47475` | `1.055` |
| `private/kimi-k2-6` | `1.5825` | `5.53875` |
The `ppq-private-mode-proxy` commit `ba984214793d3bca0f7d046b6955d42abc1c6843` changed OpenClaw display metadata to align with a 5% API margin. The live PPQ model endpoint returns the more precise rates above, which already include that margin.
Actual PPQ private billing is:
```text
price_usd =
input_tokens * input_per_1M_tokens / 1_000_000
+ output_tokens * output_per_1M_tokens / 1_000_000
```
A test request to `private/gpt-oss-120b` produced:
```json
{
"input_count": 74,
"output_count": 3,
"price_in_usd": 0.00006250875
}
```
which exactly matches:
```text
74 * 0.79125 / 1_000_000 + 3 * 1.31875 / 1_000_000
= 0.00006250875 USD
```
## Integration architectures
There are three materially different ways Routstr could integrate Tinfoil/PPQ private inference.
## Option A: User integrates Tinfoil directly
```text
User app / SDK
-> Tinfoil SecureClient / TinfoilAI
-> EHBP-encrypted request
-> Tinfoil/PPQ private enclave
```
Properties:
- Best privacy for the user.
- Routstr is not in the request path.
- User's client encrypts requests and decrypts responses.
- Usage is visible to the user's app after decryption.
- Billing is handled directly by Tinfoil/PPQ.
Example with Tinfoil's OpenAI-compatible client:
```ts
import { TinfoilAI } from "tinfoil";
const client = new TinfoilAI({
apiKey: process.env.TINFOIL_API_KEY,
transport: "ehbp",
});
const res = await client.chat.completions.create({
model: "llama3-3-70b",
messages: [{ role: "user", content: "Hello" }],
});
console.log(res.choices[0].message.content);
console.log(res.usage);
```
This is not a Routstr marketplace flow unless Routstr only acts as discovery/UI around direct Tinfoil/PPQ usage.
## Option B: Routstr integrates Tinfoil as an upstream client
```text
User -> Routstr plaintext request
-> Routstr Tinfoil SecureClient encrypts to PPQ/Tinfoil
-> PPQ private enclave
-> Routstr receives decrypted response
-> Routstr bills from decrypted usage
-> Routstr returns plaintext response to user
```
Properties:
- Easier exact billing.
- Routstr can read the decrypted OpenAI response and `usage` object.
- Routstr can charge exact PPQ token pricing.
- Privacy is different: the user sends plaintext to Routstr, and Routstr sees prompts/responses.
- End-to-end encryption is only Routstr-to-enclave, not user-to-enclave.
This should be considered a separate product/provider mode, not the same as an end-to-end private relay.
Practical implementation options:
1. Run a Node sidecar that uses the `tinfoil` npm package and expose it as a local HTTP upstream to Routstr.
2. Port EHBP client behavior to Python.
3. Reuse or adapt `ppq-private-mode-proxy` as a local upstream.
A Node sidecar is probably the quickest implementation path because `ppq-private-mode-proxy` already demonstrates the full flow.
## Option C: Routstr uses Tinfoil as a direct blind upstream
This is the current branch's design intent and is likely the best fit if Tinfoil/PPQ exposes usage metadata headers:
```text
User Tinfoil SecureClient
-> encrypted request body
-> Routstr proxy
-> Tinfoil/PPQ private API
with X-Tinfoil-Request-Usage-Metrics: true
-> PPQ private enclave
<- encrypted response
plus X-Tinfoil-Usage-Metrics / cost headers
<- Routstr proxy
-> user decrypts response
```
In this mode Routstr is still a normal upstream proxy from the user's point of view, but the upstream is Tinfoil/PPQ private inference and the body remains opaque to Routstr.
Properties:
- Strongest privacy with Routstr in the path.
- Routstr never sees plaintext prompt or plaintext response.
- Routstr can authenticate and route based on plaintext headers.
- Routstr should request Tinfoil usage metadata by adding `X-Tinfoil-Request-Usage-Metrics: true` to the upstream request.
- Tinfoil documents `X-Tinfoil-Usage-Metrics` as an upstream response header for non-streaming requests.
- For streaming requests, Tinfoil documents `X-Tinfoil-Usage-Metrics` as an HTTP trailer available only after the response body completes.
- If Tinfoil/PPQ returns `X-Tinfoil-Usage-Metrics` or a cost header on the encrypted response, Routstr can bill exactly without decrypting the body.
- If usage is only present inside the encrypted response body, Routstr still cannot read it and exact billing is not possible without a separate metadata path.
This is the only architecture that preserves end-to-end encryption from the user to the PPQ/Tinfoil enclave while still letting Routstr mediate payment. The key requirement is that usage/cost metadata must be returned outside the encrypted body, ideally as a response header available before body streaming begins.
## Current Routstr problem
The current EHBP implementation charges successful EHBP requests at `max_cost_for_model` because Routstr cannot decrypt the response body:
```text
successful EHBP request -> charge full reserved max cost
```
That is incorrect for PPQ private models. Max cost should be only a reservation/solvency ceiling. Final charge should use actual PPQ private token pricing.
Desired behavior:
```text
reserve max cost
forward encrypted request
obtain actual usage/cost metadata
finalize actual cost
refund/release the difference
```
## Usage/cost metadata requirement
For blind-relay exact billing, PPQ should return one of the following outside the encrypted response body:
```http
X-PPQ-Cost-USD: 0.00006250875
```
or:
```http
X-Private-Usage-Metrics: input=74,output=3
```
or:
```http
X-Tinfoil-Usage-Metrics: prompt=74,completion=3,total=77
```
Tinfoil proxy documentation references a usage-metrics flow where a proxy can request usage via:
```http
X-Tinfoil-Request-Usage-Metrics: true
```
and read usage from:
```http
X-Tinfoil-Usage-Metrics
```
Docs/example references:
- https://docs.tinfoil.sh/guides/proxy-server
- https://github.com/tinfoilsh/encrypted-request-proxy-example
During local PPQ testing, PPQ responses included this CORS exposure header:
```http
Access-Control-Expose-Headers: Ehbp-Response-Nonce, X-Private-Usage-Metrics, X-Encrypted-Usage-Metrics, X-Tinfoil-Usage-Metrics
```
However, the actual tested non-streaming response did not include any of these usage headers, even when `X-Tinfoil-Request-Usage-Metrics: true` was sent.
The decrypted body did include normal OpenAI usage, but only the decrypting Tinfoil client can see that body.
## Query-history fallback
PPQ's query history endpoint exposes actual usage and cost:
```text
GET https://api.ppq.ai/queries/history?page=1&page_count=...
```
A record includes:
```json
{
"timestamp": "...",
"model": "private/gpt-oss-120b",
"input_count": 74,
"output_count": 3,
"price_in_usd": 0.00006250875,
"query_type": "chat_completion",
"query_source": "api"
}
```
This could be used as a fallback, but it is less robust than response headers/trailers because matching a request to a history row can be race-prone under concurrency. It would need a reliable request identifier or metadata field that PPQ stores in history.
## Recommended Routstr direction
Use Tinfoil/PPQ as a direct blind upstream and have Routstr explicitly request usage metadata:
```text
User encrypts body
Routstr reserves max cost
Routstr forwards encrypted body to Tinfoil/PPQ /private/
Routstr includes X-Tinfoil-Request-Usage-Metrics: true
Tinfoil/PPQ returns X-Tinfoil-Usage-Metrics as a response header for non-streaming,
or as an HTTP trailer after the body completes for streaming
Routstr finalizes exact charge
User decrypts encrypted response
```
This preserves both:
- privacy: Routstr cannot read prompts/responses;
- exact billing: Routstr can charge actual PPQ private model cost, assuming Tinfoil/PPQ returns usage/cost metadata outside the encrypted body.
Implementation steps:
1. Update PPQ model fetching to include private models:
```text
GET https://api.ppq.ai/v1/models?type=all
```
2. Register `private/*` models and any Routstr-facing aliases with correct `forwarded_model_id`.
3. Keep max-cost reservation for bearer keys and X-Cashu solvency checks.
4. Add parsing support for possible usage/cost headers:
```http
X-PPQ-Cost-USD
X-Private-Usage-Metrics
X-Tinfoil-Usage-Metrics
X-Encrypted-Usage-Metrics
```
5. Finalize by actual cost instead of max cost:
```text
actual_msats = ceil((actual_usd / sats_usd_price()) * 1000)
actual_msats = max(actual_msats, settings.min_request_msat)
actual_msats = min(actual_msats, reserved_msats)
```
or, if only token counts are available:
```text
actual_usd =
input_tokens * input_per_1M_tokens / 1_000_000
+ output_tokens * output_per_1M_tokens / 1_000_000
```
6. If usage/cost metadata is missing, choose an explicit policy:
- fail closed and refund/revert;
- query PPQ history as a fallback;
- fallback to max-cost billing only if explicitly configured and clearly disclosed.
Silent max-cost billing should not be the default for PPQ private requests.
## X-Cashu consideration
For bearer-auth requests, finalization can happen after the response stream completes if usage is delivered as a trailer.
For `X-Cashu`, Routstr needs to return the refund token in the response headers. If usage/cost is only available after consuming the encrypted response stream, Routstr may need to buffer EHBP responses before sending them to the client so it can compute the refund amount first.
Possible approaches:
1. Prefer a non-trailer response header with actual cost, available before streaming body starts.
2. Buffer EHBP X-Cashu responses and then return `X-Cashu` refund.
3. Introduce a later/refund-claim mechanism, which would be a larger protocol change.
## Summary
- PPQ private models are billed per actual input/output tokens.
- Private model rates are available from `GET /v1/models?type=all`.
- Current Routstr EHBP billing at max cost is wrong for PPQ private models.
- Direct Tinfoil integration inside Routstr would enable exact usage billing but would make Routstr see plaintext.
- A blind EHBP relay preserves privacy but requires PPQ/Tinfoil to expose usage/cost in plaintext headers/trailers.
- The preferred solution is to keep Routstr blind and have PPQ return billing metadata outside the encrypted body.
## Implementation status
Direct Tinfoil upstream integration is implemented in `routstr/upstream/tinfoil.py`
and `routstr/upstream/ehbp.py`.
### What was built
- `TinfoilUpstreamProvider` (`provider_type = "tinfoil"`):
- Base URL: `https://inference.tinfoil.sh`
- Fetches models from the public `GET /v1/models` endpoint (no auth needed).
- Parses Tinfoil's pricing (`inputTokenPricePer1M`, `outputTokenPricePer1M`,
`requestPrice`) into the standard `Model`/`Pricing` schema.
- `supports_ehbp = True` — acts as a blind EHBP relay.
- `get_ehbp_forwarding_target()` returns a target that includes
`X-Tinfoil-Request-Usage-Metrics: true`.
- `forward_get_request()` proxies `/attestation` to `https://atc.tinfoil.sh/attestation`
so the SDK can fetch attestation bundles through Routstr.
- Registered in `routstr/upstream/__init__.py` and seeded from
`TINFOIL_API_KEY` env var.
- `routstr/upstream/ehbp.py`:
- `parse_tinfoil_usage_metrics()` parses
`prompt=N,completion=N[,total=N][,model=<name>]` into an OpenAI-style
usage dict. The `model` field (added in tinfoilsh/confidential-model-router
PR #385) is extracted as a string.
- `_resolve_ehbp_target_url()` overrides the forwarding URL with
`X-Tinfoil-Enclave-Url` when the SDK sends it.
- `_strip_proxy_headers()` removes `X-Routstr-Model`,
`X-Tinfoil-Enclave-Url`, and `X-Tinfoil-Request-Usage-Metrics` before
forwarding to the enclave.
- `_compute_ehbp_actual_cost()` converts the usage header into msats via
`calculate_cost()`, clamped to `[min_request_msat, max_cost_for_model]`.
When the header's `model=<name>` differs from the requested model, the
actual served model's pricing is used for cost calculation.
- `forward_ehbp_request()` (bearer auth): if `X-Tinfoil-Usage-Metrics` is
present in the response header, finalizes with `adjust_payment_for_tokens()`
for exact billing; otherwise falls back to max-cost. Billing uses the
actual served model when it differs from the requested one.
- `forward_ehbp_x_cashu_request()`: if usage is available, computes the
refund from actual cost instead of max cost, using the actual served
model's pricing when applicable.
- `routstr/proxy.py`: `/attestation` and `/tee/attestation` paths are forwarded
to Tinfoil upstreams without model/cost/auth lookups.
### Billing behavior
| Request shape | Usage source | Billing |
|---|---|---|
| Bearer, non-streaming | `X-Tinfoil-Usage-Metrics` response header | Exact token cost via `adjust_payment_for_tokens` |
| Bearer, streaming | `X-Tinfoil-Usage-Metrics` HTTP trailer | Exact token cost (h11 captures trailers) |
| Bearer, no usage header/trailer | N/A | Max-cost fallback |
| X-Cashu, non-streaming | `X-Tinfoil-Usage-Metrics` response header | Refund = `redeemed - actual_cost` |
| X-Cashu, streaming | `X-Tinfoil-Usage-Metrics` HTTP trailer | Refund = `redeemed - actual_cost` (h11 captures trailers) |
| X-Cashu, no usage header/trailer | N/A | Refund = `redeemed - max_cost` |
### Cost response headers
Since EHBP response bodies are opaque encrypted blobs, per-request cost cannot
be injected into the JSON body (as done in the normal proxy flow). Instead,
Routstr returns cost info as response headers:
| Header | Auth | Description |
|---|---|---|
| `X-Routstr-Cost-Msats` | Bearer, X-Cashu | Total msats charged for this request |
| `X-Routstr-Cost-Usd` | Bearer | USD equivalent of the charge |
| `X-Routstr-Input-Cost-Msats` | Bearer, X-Cashu | msats attributed to input tokens |
| `X-Routstr-Output-Cost-Msats` | Bearer, X-Cashu | msats attributed to output tokens |
The client/Tinfoil SDK can read these headers from the HTTP response without
needing to decrypt the body.
### Setup
```bash
TINFOIL_API_KEY=your-tinfoil-api-key
```
The provider is auto-seeded on first startup.
### Usage metrics header format
Tinfoil returns usage metrics in the `X-Tinfoil-Usage-Metrics` response header
(non-streaming) or HTTP trailer (streaming) when `X-Tinfoil-Request-Usage-Metrics:
true` is sent. As of tinfoilsh/confidential-model-router PR #385, the format is:
```
prompt=<prompt_tokens>,completion=<completion_tokens>,total=<total_tokens>,model=<served_model>
```
The `model` field carries the actual model name served by the enclave.
Routstr uses this to:
- Verify the served model matches the expected upstream model. The comparison
uses ``model_obj.forwarded_model_id`` (the actual upstream ID, e.g.
``glm-5-2``) rather than ``model_obj.id`` (the client-facing alias, e.g.
``tinfoil-glm-5-2``), so aliased models don't trigger a spurious mismatch.
- When they genuinely differ (Tinfoil served a different upstream model than
expected), look up the actual served model's pricing and use it for billing.
The reverse lookup uses ``get_model_instance``, which resolves
``forwarded_model_id`` values registered as routable aliases.
- Log the discrepancy for observability.
If the actual model is not found in Routstr's model registry, billing falls
back to the requested model's pricing.
### What still needs verification
- ~~End-to-end test with a real Tinfoil SDK client against a Routstr node with
`TINFOIL_API_KEY` set.~~ Verified: both non-streaming (header) and streaming
(trailer) responses include `model=<name>`.
- Streaming requests: usage is delivered as an HTTP trailer. Currently the
bearer path finalizes max-cost before streaming begins. Supporting streaming
usage would require buffering the response (for X-Cashu) or a deferred
finalization (for bearer).
- Whether Tinfoil's `/v1/responses` endpoint also returns usage metrics
headers.

View File

@@ -10,6 +10,7 @@ dependencies = [
"aiosqlite>=0.20",
"sqlmodel>=0.0.24",
"httpx[socks]>=0.25.2",
"h11>=0.14",
"greenlet>=3.2.1",
"alembic>=1.13",
"python-json-logger>=2.0.0",

View File

@@ -257,7 +257,20 @@ app.add_middleware(
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
expose_headers=["x-routstr-request-id", "x-cashu"],
expose_headers=[
"x-routstr-request-id",
"x-cashu",
"x-routstr-cost-msats",
"x-routstr-cost-usd",
"x-routstr-input-cost-msats",
"x-routstr-output-cost-msats",
# EHBP (Tinfoil) protocol headers must be exposed so browser clients
# can detect and decrypt encrypted responses. Without these, the
# browser hides them via CORS and the SDK treats the response as a
# plaintext proxy error, returning raw ciphertext.
"Ehbp-Response-Nonce",
"Ehbp-Encapsulated-Key",
],
)
# Add logging middleware

View File

@@ -29,6 +29,7 @@ from .payment.helpers import (
)
from .payment.models import Model
from .upstream import BaseUpstreamProvider
from .upstream.ehbp import forward_ehbp_request, forward_ehbp_x_cashu_request
from .upstream.helpers import init_upstreams
from .upstream.request_correction import correct_request, extract_error_message
@@ -104,6 +105,29 @@ def get_unique_models() -> list[Model]:
return list(_unique_models.values())
def _is_tinfoil_attestation_path(path: str) -> bool:
"""Return True for Tinfoil attestation-bundle proxy paths."""
return path in {"attestation", "tee/attestation"}
def _select_unauthenticated_get_upstreams(
path: str, upstreams: list[BaseUpstreamProvider]
) -> list[BaseUpstreamProvider]:
"""Select upstream candidates for unauthenticated GET bypass paths.
Tinfoil attestation endpoints are provider-specific. Trying every enabled
upstream can return an unrelated provider's 404 before Tinfoil is reached,
so route those paths only to Tinfoil providers.
"""
if _is_tinfoil_attestation_path(path):
return [
upstream
for upstream in upstreams
if getattr(upstream, "provider_type", None) == "tinfoil"
]
return upstreams
async def refresh_model_maps() -> None:
"""Refresh global model and provider maps using the cost-based algorithm."""
from sqlalchemy.orm import selectinload
@@ -166,6 +190,7 @@ _API_PATH_PREFIXES = (
"moderations",
"providers",
"tee/",
"attestation",
)
@@ -184,20 +209,54 @@ async def proxy(
is_responses_api = path.startswith("v1/responses") or path.startswith("responses")
request_body = await request.body()
request_body_dict = parse_request_body_json(request_body, path)
# /tee/* GET requests (e.g. attestation) don't map to models — just
# forward to all enabled upstreams without model/cost/auth lookups.
if request.method == "GET" and path.startswith("tee/"):
all_upstreams = _upstreams
# EHBP (Encrypted HTTP Body Protocol) requests carry an Ehbp-Encapsulated-Key
# header and a binary HPKE-sealed body. The proxy cannot parse the body to
# extract the model id, so the SDK sends it in X-Routstr-Model. Forward the
# raw encrypted body to the upstream's /private/ endpoint and stream the
# encrypted response back untouched — the SDK's SecureClient decrypts it.
is_ehbp = "ehbp-encapsulated-key" in headers
if is_ehbp:
request_body_dict = {}
model_id = headers.get("x-routstr-model", "")
if not model_id:
return create_error_response(
"invalid_request",
"EHBP request missing X-Routstr-Model header",
400,
request=request,
)
else:
request_body_dict = parse_request_body_json(request_body, path)
if is_responses_api:
model_id = extract_model_from_responses_request(request_body_dict)
else:
model_id = request_body_dict.get("model", "unknown")
# /tee/* and /attestation GET requests don't map to models — forward
# without model/cost/auth lookups. Tinfoil attestation paths are routed
# only to Tinfoil providers so an unrelated upstream's 404 cannot
# short-circuit before the attestation proxy is tried.
if request.method == "GET" and (
path.startswith("tee/") or path.startswith("attestation")
):
selected_upstreams = _select_unauthenticated_get_upstreams(path, _upstreams)
if not selected_upstreams:
return create_error_response(
"upstream_error",
"No upstream available for unauthenticated GET path",
502,
request=request,
)
last_error_response = None
for i, upstream in enumerate(all_upstreams):
for i, upstream in enumerate(selected_upstreams):
try:
headers = upstream.prepare_headers(dict(request.headers))
response = await upstream.forward_get_request(request, path, headers)
if response.status_code in [502, 429] and i < len(all_upstreams) - 1:
if response.status_code in [502, 429] and i < len(selected_upstreams) - 1:
logger.warning(
"Upstream %s returned %s for tee GET %s, trying next",
"Upstream %s returned %s for unauthenticated GET %s, trying next",
upstream.provider_type,
response.status_code,
path,
@@ -206,23 +265,18 @@ async def proxy(
return response
except UpstreamError as e:
logger.warning(
"Upstream %s failed for tee GET %s: %s",
"Upstream %s failed for unauthenticated GET %s: %s",
upstream.provider_type,
path,
e,
)
if i == len(all_upstreams) - 1:
if i == len(selected_upstreams) - 1:
last_error_response = create_upstream_error_response(e, request)
continue
return last_error_response or create_error_response(
"upstream_error", "All upstreams failed", 502, request=request
)
if is_responses_api:
model_id = extract_model_from_responses_request(request_body_dict)
else:
model_id = request_body_dict.get("model", "unknown")
model_obj = get_model_instance(model_id)
if not model_obj:
@@ -239,6 +293,16 @@ async def proxy(
request=request,
)
if is_ehbp:
upstreams = [upstream for upstream in upstreams if upstream.supports_ehbp]
if not upstreams:
return create_error_response(
"unsupported_request",
f"No EHBP-capable provider found for model '{model_id}'",
400,
request=request,
)
# todo figure out cost calculation since fallback provider is usually not the same price
# Use first provider for initial checks/cost calculation
# primary_upstream = upstreams[0]
@@ -258,7 +322,23 @@ async def proxy(
last_error = None
for i, upstream in enumerate(upstreams):
try:
if is_responses_api:
if is_ehbp:
if not upstream.supports_ehbp:
logger.warning(
"Upstream %s does not support EHBP for model=%s",
upstream.provider_type,
model_id,
)
continue
return await forward_ehbp_x_cashu_request(
request=request,
x_cashu_token=x_cashu,
path=path,
max_cost_for_model=max_cost_for_model,
model_obj=model_obj,
upstream=upstream,
)
elif is_responses_api:
return await upstream.handle_x_cashu_responses(
request, x_cashu, path, max_cost_for_model, model_obj
)
@@ -347,7 +427,7 @@ async def proxy(
"upstream_error", "All upstreams failed", 502, request=request
)
if request_body_dict:
if is_ehbp or request_body_dict:
await pay_for_request(key, max_cost_for_model, session)
# Tracks request params already removed in response to upstream rejections,
@@ -361,7 +441,29 @@ async def proxy(
try:
while True:
try:
if is_responses_api:
if is_ehbp:
if not upstream.supports_ehbp:
logger.warning(
"Upstream %s does not support EHBP for model=%s",
upstream.provider_type,
model_id,
)
raise UpstreamError(
f"Provider {upstream.provider_type} does not support EHBP",
status_code=400,
)
response = await forward_ehbp_request(
request=request,
path=path,
headers=headers,
request_body=request_body,
upstream=upstream,
key=key,
max_cost_for_model=max_cost_for_model,
session=session,
model_obj=model_obj,
)
elif is_responses_api:
response = await upstream.forward_responses_request(
request,
path,
@@ -406,7 +508,7 @@ async def proxy(
# When the upstream 400s naming such a param, strip it from the
# body and retry the SAME upstream. ``already_stripped`` bounds
# this to one retry per distinct param so it always terminates.
if response.status_code == 400:
if response.status_code == 400 and not is_ehbp:
correction = correct_request(
request_body,
extract_error_message(response),

View File

@@ -11,6 +11,7 @@ from .openrouter import OpenRouterUpstreamProvider
from .perplexity import PerplexityUpstreamProvider
from .ppqai import PPQAIUpstreamProvider
from .routstr import RoutstrUpstreamProvider
from .tinfoil import TinfoilUpstreamProvider
from .xai import XAIUpstreamProvider
upstream_provider_classes: list[type[BaseUpstreamProvider]] = [
@@ -26,6 +27,7 @@ upstream_provider_classes: list[type[BaseUpstreamProvider]] = [
PerplexityUpstreamProvider,
PPQAIUpstreamProvider,
RoutstrUpstreamProvider,
TinfoilUpstreamProvider,
XAIUpstreamProvider,
]
"""List of all upstream classes"""

View File

@@ -4,6 +4,7 @@ import asyncio
import json
import math
import traceback
import typing
import uuid
from collections.abc import AsyncGenerator, AsyncIterator, Iterator
from typing import Any, Mapping, Self, cast
@@ -55,6 +56,9 @@ from .count_tokens import count_tokens_locally
from .litellm_routing import detect_litellm_prefix
from .rate_limit import UPSTREAM_RATE_LIMIT, classify_rate_limit
if typing.TYPE_CHECKING:
from .ehbp import ConfidentialInferenceProfile, EHBPForwardingTarget
logger = get_logger(__name__)
@@ -2845,6 +2849,26 @@ class BaseUpstreamProvider:
# Don't revert here — proxy.py owns payment revert to avoid double-revert
raise UpstreamError("An unexpected server error occurred", status_code=500)
supports_ehbp: bool = False
def get_confidential_inference_profile(self) -> "ConfidentialInferenceProfile | None":
"""Return provider policy for encrypted/confidential inference forwarding."""
return None
def get_ehbp_forwarding_target(
self, path: str, model_obj: Model
) -> "EHBPForwardingTarget":
"""Return the EHBP forwarding target for this provider.
Providers must explicitly opt in by setting ``supports_ehbp = True``
and overriding this method. Most upstreams do not accept EHBP-encrypted
request bodies, so the base provider intentionally does not provide a
default endpoint.
"""
raise NotImplementedError(
f"Provider {self.provider_type} does not support EHBP forwarding"
)
async def forward_responses_request(
self,
request: Request,

1200
routstr/upstream/ehbp.py Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -263,6 +263,7 @@ async def _seed_providers_from_settings(
("PERPLEXITY_API_KEY", "perplexity", None, None),
("FIREWORKS_API_KEY", "fireworks", None, None),
("XAI_API_KEY", "xai", None, None),
("TINFOIL_API_KEY", "tinfoil", None, None),
]
for env_key, provider_type, _, _ in env_mappings:

View File

@@ -8,6 +8,7 @@ from pydantic.v1 import BaseModel, Field
from ..core.logging import get_logger
from ..payment.models import Architecture, Model, Pricing, async_fetch_openrouter_models
from .base import BaseUpstreamProvider, TopupData
from .ehbp import EHBPForwardingTarget
if TYPE_CHECKING:
from ..core.db import UpstreamProviderRow
@@ -39,6 +40,10 @@ class PPQAIUpstreamProvider(BaseUpstreamProvider):
default_base_url = "https://api.ppq.ai"
platform_url = "https://ppq.ai/api-docs"
IGNORED_MODEL_IDS: list[str] = ["auto"]
# PPQ.AI has a private encrypted endpoint, but this proxy currently has no
# provider-attested usage extractor/model binding for it. Keep EHBP disabled
# until a ConfidentialInferenceProfile can bill it without max-cost fallback.
supports_ehbp = False
def __init__(self, api_key: str, provider_fee: float = 1.0):
super().__init__(
@@ -70,6 +75,20 @@ class PPQAIUpstreamProvider(BaseUpstreamProvider):
def transform_model_name(self, model_id: str) -> str:
return model_id
def get_ehbp_forwarding_target(
self, path: str, model_obj: Model
) -> EHBPForwardingTarget:
"""Return the PPQ.AI private enclave target for EHBP requests.
PPQ.AI exposes EHBP-aware inference under /private/v1/... separate
from the public /v1/... endpoint. The encrypted body remains opaque to
Routstr, so PPQ.AI also needs X-Private-Model for routing/billing.
"""
return EHBPForwardingTarget(
url=f"{self.base_url.rstrip('/')}/private/{path.lstrip('/')}",
headers={"X-Private-Model": model_obj.forwarded_model_id or model_obj.id},
)
@classmethod
async def create_account_static(cls) -> dict[str, object]:
"""Create a new PPQ.AI account without requiring an instance.

236
routstr/upstream/tinfoil.py Normal file
View File

@@ -0,0 +1,236 @@
from __future__ import annotations
from typing import TYPE_CHECKING
import httpx
from fastapi import Request
from fastapi.responses import Response, StreamingResponse
from pydantic.v1 import BaseModel
from ..core.exceptions import UpstreamError
from ..core.logging import get_logger
from ..payment.models import Architecture, Model, Pricing
from .base import BaseUpstreamProvider
from .ehbp import (
_ENCLAVE_URL_HEADER,
_PROXY_ONLY_HEADERS,
_RESPONSE_USAGE_HEADER,
ConfidentialInferenceProfile,
EHBPForwardingTarget,
)
if TYPE_CHECKING:
from ..core.db import UpstreamProviderRow
logger = get_logger(__name__)
class TinfoilModelPricing(BaseModel):
inputTokenPricePer1M: float = 0.0
outputTokenPricePer1M: float = 0.0
requestPrice: float = 0.0
class TinfoilModel(BaseModel):
id: str
context_window: int = 0
created: int = 0
multimodal: bool = False
reasoning: bool = False
tool_calling: bool = False
type: str = "chat"
pricing: TinfoilModelPricing = TinfoilModelPricing()
endpoints: list[str] = []
class TinfoilUpstreamProvider(BaseUpstreamProvider):
"""Direct upstream provider for the Tinfoil inference API.
Tinfoil hosts open-source models inside attested secure enclaves and exposes
an OpenAI-compatible API at ``https://inference.tinfoil.sh``. Request and
response bodies are encrypted end-to-end with EHBP (HPKE), so Routstr acts
as a blind relay: it forwards the opaque encrypted body, never sees
plaintext, and bills from the ``X-Tinfoil-Usage-Metrics`` header that
Tinfoil returns outside the encrypted body when
``X-Tinfoil-Request-Usage-Metrics: true`` is set.
"""
provider_type = "tinfoil"
default_base_url = "https://inference.tinfoil.sh"
platform_url = "https://docs.tinfoil.sh"
supports_ehbp = True
confidential_inference_profile = ConfidentialInferenceProfile(
usage_response_header=_RESPONSE_USAGE_HEADER,
client_target_url_header=_ENCLAVE_URL_HEADER,
allow_client_target_override=True,
proxy_only_headers=_PROXY_ONLY_HEADERS,
)
def __init__(self, api_key: str, provider_fee: float = 1.0):
super().__init__(
base_url=self.default_base_url,
api_key=api_key,
provider_fee=provider_fee,
)
@classmethod
def _build_from_row(
cls, provider_row: "UpstreamProviderRow"
) -> "TinfoilUpstreamProvider":
return cls(
api_key=provider_row.api_key,
provider_fee=provider_row.provider_fee,
)
@classmethod
def get_provider_metadata(cls) -> dict[str, object]:
return {
"id": cls.provider_type,
"name": "Tinfoil",
"default_base_url": cls.default_base_url,
"fixed_base_url": True,
"platform_url": cls.platform_url,
"can_create_account": False,
"can_topup": False,
"can_show_balance": False,
}
def transform_model_name(self, model_id: str) -> str:
return model_id.removeprefix("tinfoil/")
def get_confidential_inference_profile(self) -> ConfidentialInferenceProfile:
return self.confidential_inference_profile
async def forward_get_request(
self,
request: Request,
path: str,
headers: dict,
) -> Response | StreamingResponse:
"""Handle Tinfoil-specific GET endpoints.
* ``/attestation`` (or ``/tee/attestation``): proxy to the Tinfoil ATC
(attestation bundle proxy) at ``https://atc.tinfoil.sh/attestation``.
* Other GETs: forward to the provider base URL
(``https://inference.tinfoil.sh``). ``X-Tinfoil-Enclave-Url`` is an
EHBP-only header used for encrypted POST requests and is not honored
for unencrypted GET requests.
"""
clean_path = path.removeprefix("tee/")
if clean_path == "attestation":
return await self._proxy_attestation(headers)
return await super().forward_get_request(request, path, headers)
async def _proxy_attestation(self, headers: dict) -> Response:
url = "https://atc.tinfoil.sh/attestation"
async with httpx.AsyncClient(
transport=httpx.AsyncHTTPTransport(retries=1),
timeout=30.0,
) as client:
try:
resp = await client.get(
url,
headers={
"Accept": headers.get("accept", "application/json"),
},
)
response_headers = dict(resp.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
return Response(
content=resp.content,
status_code=resp.status_code,
headers=response_headers,
)
except Exception as exc:
raise UpstreamError(
f"Error fetching Tinfoil attestation: {type(exc).__name__}",
status_code=502,
) from exc
def get_ehbp_forwarding_target(
self, path: str, model_obj: Model
) -> EHBPForwardingTarget:
"""Return the Tinfoil enclave target for EHBP requests.
Requests usage metrics from the enclave so Routstr can bill exactly
without decrypting the response body. The actual forwarding URL is
overridden at dispatch time by ``X-Tinfoil-Enclave-Url`` when the SDK
sends it (see ``routstr/upstream/ehbp.py``).
"""
return EHBPForwardingTarget(
url=f"{self.base_url.rstrip('/')}/{path.lstrip('/')}",
headers={"X-Tinfoil-Request-Usage-Metrics": "true"},
profile=self.confidential_inference_profile,
)
async def fetch_models(self) -> list[Model]:
"""Fetch models from the public Tinfoil models endpoint.
``GET /v1/models`` is unauthenticated and returns all available models
with their pricing in USD per 1M tokens.
"""
url = f"{self.base_url}/v1/models"
try:
async with httpx.AsyncClient(timeout=30.0) as client:
response = await client.get(url)
response.raise_for_status()
data = response.json()
models_data = data.get("data", [])
models: list[Model] = []
for model_data in models_data:
try:
tf = TinfoilModel.parse_obj(model_data)
input_price = tf.pricing.inputTokenPricePer1M
output_price = tf.pricing.outputTokenPricePer1M
request_price = tf.pricing.requestPrice
modality = "text->text"
input_modalities = ["text"]
output_modalities = ["text"]
if tf.multimodal:
modality = "text->text+image"
input_modalities = ["text", "image"]
models.append(
Model(
id=tf.id,
name=tf.id,
created=tf.created,
description=f"Tinfoil {tf.type} model",
context_length=tf.context_window,
architecture=Architecture(
modality=modality,
input_modalities=input_modalities,
output_modalities=output_modalities,
tokenizer="Unknown",
instruct_type=None,
),
pricing=Pricing(
prompt=input_price / 1_000_000,
completion=output_price / 1_000_000,
request=request_price,
image=0.0,
web_search=0.0,
internal_reasoning=0.0,
),
)
)
except Exception as e:
logger.warning(
"Failed to parse Tinfoil model",
extra={
"model_id": model_data.get("id", "unknown"),
"error": str(e),
"error_type": type(e).__name__,
},
)
return models
except Exception as e:
logger.error(
"Error fetching models from Tinfoil",
extra={"error": str(e), "error_type": type(e).__name__},
)
return []

View File

@@ -0,0 +1,161 @@
"""h11-based HTTP client for EHBP requests that captures HTTP trailers.
httpx/httpcore silently discard HTTP trailers during chunked transfer
decoding. Tinfoil returns ``X-Tinfoil-Usage-Metrics`` as a trailer on
streaming responses, so we need a lower-level HTTP client that preserves
trailers from the h11 ``EndOfMessage`` event.
Because EHBP response bodies are opaque encrypted blobs, buffering the full
response is acceptable — the client decrypts the complete body regardless of
whether it arrived streamed or buffered.
"""
from __future__ import annotations
import asyncio
import ssl
from dataclasses import dataclass, field
from urllib.parse import urlsplit
import h11
from ..core import get_logger
logger = get_logger(__name__)
_READ_BUFSIZE = 65536
_DEFAULT_TIMEOUT_SECONDS = 30.0
_DEFAULT_CLOSE_TIMEOUT_SECONDS = 1.0
_DEFAULT_MAX_RESPONSE_BYTES = 25 * 1024 * 1024
@dataclass
class TrailerResponse:
"""Buffered HTTP response with optional trailer headers."""
status_code: int
headers: list[tuple[str, str]]
body: bytes
trailers: list[tuple[str, str]] = field(default_factory=list)
def _get_header(headers: list[tuple[str, str]], name: str) -> str | None:
name_lower = name.lower()
for k, v in headers:
if k.lower() == name_lower:
return v
return None
async def forward_with_trailer(
*,
method: str,
url: str,
headers: dict[str, str],
body: bytes,
timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS,
max_response_bytes: int = _DEFAULT_MAX_RESPONSE_BYTES,
close_timeout_seconds: float = _DEFAULT_CLOSE_TIMEOUT_SECONDS,
) -> TrailerResponse:
"""Send an HTTP/1.1 request via h11 and capture HTTP trailers.
Returns a :class:`TrailerResponse` with the full buffered body and any
trailer headers from the ``EndOfMessage`` event.
"""
parsed = urlsplit(url)
host = parsed.hostname
if not host:
raise ValueError(f"Invalid URL (no hostname): {url}")
port = parsed.port or 443
path = parsed.path or "/"
if parsed.query:
path = f"{path}?{parsed.query}"
ssl_ctx = ssl.create_default_context()
reader, writer = await asyncio.wait_for(
asyncio.open_connection(host, port, ssl=ssl_ctx),
timeout=timeout_seconds,
)
try:
# Build HTTP/1.1 request
header_lines = [f"{method} {path} HTTP/1.1"]
has_host = any(k.lower() == "host" for k in headers)
if not has_host:
header_lines.append(f"Host: {host}")
header_lines.append("Connection: close")
for key, value in headers.items():
if key.lower() in ("host", "connection"):
continue
header_lines.append(f"{key}: {value}")
if body and not any(k.lower() == "content-length" for k in headers):
header_lines.append(f"Content-Length: {len(body)}")
request_data = "\r\n".join(header_lines).encode() + b"\r\n\r\n"
if body:
request_data += body
writer.write(request_data)
await asyncio.wait_for(writer.drain(), timeout=timeout_seconds)
# Parse response with h11
conn = h11.Connection(h11.CLIENT)
status_code = 0
resp_headers: list[tuple[str, str]] = []
body_chunks: list[bytes] = []
body_size = 0
trailers: list[tuple[str, str]] = []
while True:
event = conn.next_event()
if event is h11.NEED_DATA:
data = await asyncio.wait_for(
reader.read(_READ_BUFSIZE),
timeout=timeout_seconds,
)
conn.receive_data(data if data else b"")
continue
if isinstance(event, h11.Response):
status_code = event.status_code
resp_headers = [(k.decode(), v.decode()) for k, v in event.headers]
elif isinstance(event, h11.Data):
body_size += len(event.data)
if body_size > max_response_bytes:
raise ValueError(
f"EHBP response exceeded {max_response_bytes} bytes"
)
body_chunks.append(event.data)
elif isinstance(event, h11.EndOfMessage):
for k, v in event.headers:
trailers.append((k.decode(), v.decode()))
break
elif isinstance(event, h11.PAUSED):
# Shouldn't happen for simple request/response, but break safely
logger.warning("h11 PAUSED event during EHBP response parsing")
break
elif isinstance(event, h11.ConnectionClosed):
break
return TrailerResponse(
status_code=status_code,
headers=resp_headers,
body=b"".join(body_chunks),
trailers=trailers,
)
finally:
writer.close()
if close_timeout_seconds > 0:
try:
await asyncio.wait_for(
writer.wait_closed(), timeout=close_timeout_seconds
)
except Exception:
pass

View File

@@ -0,0 +1,185 @@
from __future__ import annotations
from typing import AsyncGenerator
import pytest
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from sqlalchemy.pool import StaticPool
from sqlmodel import SQLModel, select
from sqlmodel.ext.asyncio.session import AsyncSession
from routstr.core.db import ApiKey
from routstr.upstream.ehbp import (
finalize_ehbp_actual_cost_payment,
finalize_ehbp_max_cost_payment,
)
def _make_engine() -> AsyncEngine:
return create_async_engine(
"sqlite+aiosqlite://",
poolclass=StaticPool,
connect_args={"check_same_thread": False},
)
@pytest.fixture
async def session(monkeypatch: pytest.MonkeyPatch) -> AsyncGenerator[AsyncSession, None]:
monkeypatch.setattr("routstr.upstream.ehbp.ROUTSTR_FEE_PERCENT", 0)
engine = _make_engine()
async with engine.begin() as conn:
await conn.run_sync(SQLModel.metadata.create_all)
db_session = AsyncSession(engine, expire_on_commit=False)
try:
yield db_session
finally:
await db_session.close()
await engine.dispose()
async def _api_key(session: AsyncSession, hashed_key: str) -> ApiKey | None:
return (
await session.exec(select(ApiKey).where(ApiKey.hashed_key == hashed_key))
).one_or_none()
@pytest.mark.asyncio
async def test_finalize_actual_cost_payment_updates_balance_and_releases_reserve(
session: AsyncSession,
) -> None:
key = ApiKey(
hashed_key="ehbp-actual",
balance=10_000,
reserved_balance=3_000,
reserved_at=123,
)
session.add(key)
await session.commit()
await finalize_ehbp_actual_cost_payment(
key,
session,
reserved_cost_for_model=3_000,
model_id="tinfoil/model",
cost_info={
"total_msats": 1_200,
"input_tokens": 10,
"output_tokens": 20,
"input_msats": 500,
"output_msats": 700,
},
)
updated = await _api_key(session, "ehbp-actual")
assert updated is not None
assert updated.balance == 8_800
assert updated.reserved_balance == 0
assert updated.reserved_at is None
assert updated.total_spent == 1_200
@pytest.mark.asyncio
async def test_finalize_max_cost_payment_updates_parent_and_child_spend(
session: AsyncSession,
) -> None:
parent = ApiKey(
hashed_key="ehbp-parent",
balance=10_000,
reserved_balance=3_000,
reserved_at=123,
)
child = ApiKey(
hashed_key="ehbp-child",
balance=0,
reserved_balance=3_000,
reserved_at=123,
parent_key_hash="ehbp-parent",
)
session.add(parent)
session.add(child)
await session.commit()
await finalize_ehbp_max_cost_payment(
child,
session,
max_cost_for_model=3_000,
model_id="tinfoil/model",
)
updated_parent = await _api_key(session, "ehbp-parent")
updated_child = await _api_key(session, "ehbp-child")
assert updated_parent is not None
assert updated_child is not None
assert updated_parent.balance == 7_000
assert updated_parent.reserved_balance == 0
assert updated_parent.reserved_at is None
assert updated_parent.total_spent == 3_000
assert updated_child.balance == 0
assert updated_child.reserved_balance == 0
assert updated_child.reserved_at is None
assert updated_child.total_spent == 3_000
@pytest.mark.asyncio
async def test_finalize_actual_cost_payment_rolls_back_when_parent_update_matches_no_rows(
session: AsyncSession,
) -> None:
key = ApiKey(
hashed_key="ehbp-missing-parent",
balance=10_000,
reserved_balance=3_000,
reserved_at=123,
)
session.add(key)
await session.commit()
await session.delete(key)
await session.commit()
await finalize_ehbp_actual_cost_payment(
key,
session,
reserved_cost_for_model=3_000,
model_id="tinfoil/model",
cost_info={"total_msats": 1_200},
)
assert await _api_key(session, "ehbp-missing-parent") is None
@pytest.mark.asyncio
async def test_finalize_max_cost_payment_rolls_back_parent_when_child_update_matches_no_rows(
session: AsyncSession,
) -> None:
parent = ApiKey(
hashed_key="ehbp-rollback-parent",
balance=10_000,
reserved_balance=3_000,
reserved_at=123,
)
child = ApiKey(
hashed_key="ehbp-missing-child",
balance=0,
reserved_balance=3_000,
reserved_at=123,
parent_key_hash="ehbp-rollback-parent",
)
session.add(parent)
session.add(child)
await session.commit()
await session.delete(child)
await session.commit()
await finalize_ehbp_max_cost_payment(
child,
session,
max_cost_for_model=3_000,
model_id="tinfoil/model",
)
updated_parent = await _api_key(session, "ehbp-rollback-parent")
assert updated_parent is not None
assert updated_parent.balance == 10_000
assert updated_parent.reserved_balance == 3_000
assert updated_parent.reserved_at == 123
assert updated_parent.total_spent == 0
assert await _api_key(session, "ehbp-missing-child") is None

View File

@@ -0,0 +1,95 @@
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import FastAPI
from fastapi.responses import Response
from httpx import ASGITransport, AsyncClient
from routstr import proxy as proxy_module
@pytest.fixture
def proxy_app() -> FastAPI:
app = FastAPI()
app.include_router(proxy_module.proxy_router)
return app
@pytest.mark.asyncio
async def test_attestation_get_routes_directly_to_tinfoil_provider(
monkeypatch: pytest.MonkeyPatch, proxy_app: FastAPI
) -> None:
non_tinfoil = MagicMock()
non_tinfoil.provider_type = "openai"
non_tinfoil.prepare_headers = MagicMock(return_value={})
non_tinfoil.forward_get_request = AsyncMock(
return_value=Response(status_code=404, content=b"wrong upstream")
)
tinfoil = MagicMock()
tinfoil.provider_type = "tinfoil"
tinfoil.prepare_headers = MagicMock(return_value={"accept": "application/json"})
tinfoil.forward_get_request = AsyncMock(
return_value=Response(status_code=200, content=b'{"attestation":true}')
)
monkeypatch.setattr(proxy_module, "_upstreams", [non_tinfoil, tinfoil])
async with AsyncClient(
transport=ASGITransport(app=proxy_app), base_url="http://test" # type: ignore[arg-type]
) as client:
response = await client.get("/attestation")
assert response.status_code == 200
assert response.content == b'{"attestation":true}'
non_tinfoil.forward_get_request.assert_not_called()
tinfoil.forward_get_request.assert_awaited_once()
@pytest.mark.asyncio
async def test_tee_attestation_get_routes_directly_to_tinfoil_provider(
monkeypatch: pytest.MonkeyPatch, proxy_app: FastAPI
) -> None:
non_tinfoil = MagicMock()
non_tinfoil.provider_type = "openrouter"
non_tinfoil.prepare_headers = MagicMock(return_value={})
non_tinfoil.forward_get_request = AsyncMock(
return_value=Response(status_code=404, content=b"wrong upstream")
)
tinfoil = MagicMock()
tinfoil.provider_type = "tinfoil"
tinfoil.prepare_headers = MagicMock(return_value={"accept": "application/json"})
tinfoil.forward_get_request = AsyncMock(
return_value=Response(status_code=200, content=b'{"tee":true}')
)
monkeypatch.setattr(proxy_module, "_upstreams", [non_tinfoil, tinfoil])
async with AsyncClient(
transport=ASGITransport(app=proxy_app), base_url="http://test" # type: ignore[arg-type]
) as client:
response = await client.get("/tee/attestation")
assert response.status_code == 200
assert response.content == b'{"tee":true}'
non_tinfoil.forward_get_request.assert_not_called()
tinfoil.forward_get_request.assert_awaited_once()
def test_attestation_upstream_selection_is_tinfoil_only() -> None:
non_tinfoil = MagicMock(provider_type="openai")
tinfoil = MagicMock(provider_type="tinfoil")
assert proxy_module._select_unauthenticated_get_upstreams(
"attestation", [non_tinfoil, tinfoil]
) == [tinfoil]
assert proxy_module._select_unauthenticated_get_upstreams(
"tee/attestation", [non_tinfoil, tinfoil]
) == [tinfoil]
assert proxy_module._select_unauthenticated_get_upstreams(
"tee/other", [non_tinfoil, tinfoil]
) == [non_tinfoil, tinfoil]

View File

@@ -0,0 +1,749 @@
"""Unit tests for Tinfoil direct integration.
Covers the EHBP usage-metrics header parser, the proxy header stripping, the
enclave URL override, and the TinfoilUpstreamProvider model fetching/forwarding
target logic.
"""
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from routstr.upstream.ehbp import (
_PROXY_ONLY_HEADERS,
_compute_ehbp_actual_cost,
_prepare_ehbp_upstream_headers,
_resolve_ehbp_target_url,
_strip_proxy_headers,
parse_tinfoil_usage_metrics,
)
from routstr.upstream.tinfoil import (
TinfoilModel,
TinfoilUpstreamProvider,
)
# ---------------------------------------------------------------------------
# parse_tinfoil_usage_metrics
# ---------------------------------------------------------------------------
class TestParseTinfoilUsageMetrics:
def test_full_header(self) -> None:
result = parse_tinfoil_usage_metrics("prompt=67,completion=42,total=109")
assert result == {
"prompt_tokens": 67,
"completion_tokens": 42,
"total_tokens": 109,
}
def test_without_total(self) -> None:
result = parse_tinfoil_usage_metrics("prompt=10,completion=5")
assert result == {"prompt_tokens": 10, "completion_tokens": 5}
def test_none(self) -> None:
assert parse_tinfoil_usage_metrics(None) is None
def test_empty(self) -> None:
assert parse_tinfoil_usage_metrics("") is None
def test_malformed(self) -> None:
assert parse_tinfoil_usage_metrics("garbage") is None
def test_missing_completion(self) -> None:
assert parse_tinfoil_usage_metrics("prompt=10") is None
def test_extra_whitespace(self) -> None:
result = parse_tinfoil_usage_metrics(
"prompt = 100 , completion = 200 , total = 300"
)
assert result == {
"prompt_tokens": 100,
"completion_tokens": 200,
"total_tokens": 300,
}
def test_with_model_field(self) -> None:
result = parse_tinfoil_usage_metrics(
"prompt=42,completion=10,total=52,model=llama3-3-70b"
)
assert result == {
"prompt_tokens": 42,
"completion_tokens": 10,
"total_tokens": 52,
"model": "llama3-3-70b",
}
def test_with_model_no_total(self) -> None:
result = parse_tinfoil_usage_metrics(
"prompt=67,completion=42,model=gpt-oss-120b"
)
assert result == {
"prompt_tokens": 67,
"completion_tokens": 42,
"model": "gpt-oss-120b",
}
def test_model_with_dashes_and_numbers(self) -> None:
result = parse_tinfoil_usage_metrics(
"prompt=1,completion=1,total=2,model=kimi-k2-6"
)
assert result is not None
assert result["model"] == "kimi-k2-6"
def test_model_with_extra_fields(self) -> None:
result = parse_tinfoil_usage_metrics(
"prompt=69,completion=20,total=89,"
"cached_prompt_tokens=64,uncached_prompt_tokens=5,"
"model=kimi-k2-6"
)
assert result is not None
assert result["prompt_tokens"] == 69
assert result["completion_tokens"] == 20
assert result["total_tokens"] == 89
assert result["model"] == "kimi-k2-6"
def test_old_format_still_works(self) -> None:
"""Headers without the model field (pre-PR #385) still parse."""
result = parse_tinfoil_usage_metrics(
"prompt=67,completion=42,total=109"
)
assert result == {
"prompt_tokens": 67,
"completion_tokens": 42,
"total_tokens": 109,
}
assert "model" not in result
# ---------------------------------------------------------------------------
# _strip_proxy_headers
# ---------------------------------------------------------------------------
class TestStripProxyHeaders:
def test_strips_all_proxy_only(self) -> None:
headers = {
"x-routstr-model": "tinfoil-llama3-3-70b",
"X-Tinfoil-Enclave-Url": "https://inference.tinfoil.sh",
"X-Tinfoil-Request-Usage-Metrics": "true",
"Authorization": "Bearer secret",
"Ehbp-Encapsulated-Key": "abc123",
}
clean = _strip_proxy_headers(headers)
assert "x-routstr-model" not in clean
assert "X-Tinfoil-Enclave-Url" not in clean
assert "X-Tinfoil-Request-Usage-Metrics" not in clean
assert clean["Authorization"] == "Bearer secret"
assert clean["Ehbp-Encapsulated-Key"] == "abc123"
def test_all_proxy_only_headers_covered(self) -> None:
assert _PROXY_ONLY_HEADERS == {
"x-routstr-model",
"x-tinfoil-enclave-url",
"x-tinfoil-request-usage-metrics",
}
class TestPrepareEHBPUpstreamHeaders:
def test_strips_client_proxy_headers_before_merging_target_headers(self) -> None:
headers = {
"x-routstr-model": "tinfoil-llama3-3-70b",
"X-Tinfoil-Enclave-Url": "https://enclave.tinfoil.sh",
"X-Tinfoil-Request-Usage-Metrics": "false",
"Authorization": "Bearer upstream-key",
"Ehbp-Encapsulated-Key": "abc123",
}
target_headers = {"X-Tinfoil-Request-Usage-Metrics": "true"}
clean = _prepare_ehbp_upstream_headers(headers, target_headers)
assert "x-routstr-model" not in clean
assert "X-Tinfoil-Enclave-Url" not in clean
assert clean["Authorization"] == "Bearer upstream-key"
assert clean["Ehbp-Encapsulated-Key"] == "abc123"
assert clean["X-Tinfoil-Request-Usage-Metrics"] == "true"
# ---------------------------------------------------------------------------
# _resolve_ehbp_target_url
# ---------------------------------------------------------------------------
class TestResolveEhbpTargetUrl:
def test_override_with_enclave_url_for_tinfoil(self) -> None:
result = _resolve_ehbp_target_url(
"https://default.example.com/v1/chat/completions",
"v1/chat/completions",
{"X-Tinfoil-Enclave-Url": "https://enclave.tinfoil.sh"},
"tinfoil",
)
assert result == "https://enclave.tinfoil.sh/v1/chat/completions"
def test_override_lowercase_header_for_tinfoil(self) -> None:
result = _resolve_ehbp_target_url(
"https://default.example.com/v1/chat/completions",
"v1/chat/completions",
{"x-tinfoil-enclave-url": "https://enclave.tinfoil.sh"},
"tinfoil",
)
assert result == "https://enclave.tinfoil.sh/v1/chat/completions"
def test_no_override(self) -> None:
default = "https://inference.tinfoil.sh/v1/chat/completions"
result = _resolve_ehbp_target_url(
default,
"v1/chat/completions",
{},
"tinfoil",
)
assert result == default
def test_non_tinfoil_provider_ignores_enclave_url(self) -> None:
default = "https://api.ppq.ai/private/v1/chat/completions"
result = _resolve_ehbp_target_url(
default,
"v1/chat/completions",
{"X-Tinfoil-Enclave-Url": "https://enclave.tinfoil.sh"},
"ppqai",
)
assert result == default
@pytest.mark.parametrize(
"bad_url",
[
"http://enclave.tinfoil.sh",
"https://attacker.example",
"https://tinfoil.sh.attacker.example",
"https://127.0.0.1",
"https://enclave.tinfoil.sh:8443",
"https://user:pass@enclave.tinfoil.sh",
],
)
def test_tinfoil_rejects_unsafe_enclave_url(self, bad_url: str) -> None:
from routstr.core.exceptions import UpstreamError
with pytest.raises(UpstreamError):
_resolve_ehbp_target_url(
"https://default.example.com/v1/chat/completions",
"v1/chat/completions",
{"X-Tinfoil-Enclave-Url": bad_url},
"tinfoil",
)
# ---------------------------------------------------------------------------
# _compute_ehbp_actual_cost
# ---------------------------------------------------------------------------
class TestComputeEhbpActualCost:
@pytest.mark.asyncio
async def test_no_usage_falls_back_to_max_cost(self) -> None:
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
result = await _compute_ehbp_actual_cost(None, model_obj, 100_000)
assert result["total_msats"] == 100_000
assert result["input_tokens"] == 0
assert result["output_tokens"] == 0
@pytest.mark.asyncio
async def test_usage_parsed_and_clamped(self) -> None:
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
# The actual cost from calculate_cost will be small; we just verify
# it's clamped to min_request_msat at minimum.
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=10,
output_msats=20,
total_msats=30,
total_usd=0.0001,
input_tokens=67,
output_tokens=42,
)
result = await _compute_ehbp_actual_cost(
"prompt=67,completion=42,total=109",
model_obj,
100_000,
)
assert result["total_msats"] == 30
assert result["total_msats"] <= 100_000
assert result["input_tokens"] == 67
assert result["output_tokens"] == 42
assert result["total_tokens"] == 109
assert result["input_msats"] == 10
assert result["output_msats"] == 20
@pytest.mark.asyncio
async def test_max_cost_data_falls_back(self) -> None:
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import MaxCostData
mock_calc.return_value = MaxCostData(
base_msats=0,
input_msats=0,
output_msats=0,
total_msats=0,
total_usd=0.0,
input_tokens=0,
output_tokens=0,
)
result = await _compute_ehbp_actual_cost(
"prompt=0,completion=0",
model_obj,
50_000,
)
assert result["total_msats"] == 50_000
assert result["input_tokens"] == 0
assert result["output_tokens"] == 0
@pytest.mark.asyncio
async def test_model_match_no_actual_model_key(self) -> None:
"""When the served model matches the requested one, no actual_model key."""
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=llama3-3-70b",
model_obj,
100_000,
)
assert "actual_model" not in result
# calculate_cost called with requested model
call_args = mock_calc.call_args
assert call_args[0][0]["model"] == "llama3-3-70b"
@pytest.mark.asyncio
async def test_alias_match_no_actual_model_key(self) -> None:
"""When the served upstream model matches forwarded_model_id through
a client-facing alias, no actual_model key is set."""
model_obj = MagicMock()
model_obj.id = "tinfoil-glm-5-2" # client-facing alias
model_obj.forwarded_model_id = "glm-5-2" # actual upstream ID
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
# Tinfoil header returns the actual upstream model ID
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=glm-5-2",
model_obj,
100_000,
)
assert "actual_model" not in result
# calculate_cost called with the client-facing model ID (whose
# pricing includes the correct upstream rates)
call_args = mock_calc.call_args
assert call_args[0][0]["model"] == "tinfoil-glm-5-2"
@pytest.mark.asyncio
async def test_real_mismatch_uses_actual_model_for_pricing(self) -> None:
"""When the served model differs from the expected upstream model,
the actual model's pricing is used."""
model_obj = MagicMock()
model_obj.id = "tinfoil-gpt-oss-120b" # client-facing alias
model_obj.forwarded_model_id = "gpt-oss-120b" # expected upstream
actual_model_obj = MagicMock()
actual_model_obj.id = "tinfoil-llama3-3-70b" # client-facing of actual
actual_model_obj.forwarded_model_id = "llama3-3-70b"
with patch(
"routstr.proxy.get_model_instance",
return_value=actual_model_obj,
), patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=20,
output_msats=40,
total_msats=60,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
# Tinfoil served llama3-3-70b instead of gpt-oss-120b
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=llama3-3-70b",
model_obj,
100_000,
)
assert result["actual_model"] == "llama3-3-70b"
assert result["total_msats"] == 60
# calculate_cost called with the actual model's client-facing ID
call_args = mock_calc.call_args
assert call_args[0][0]["model"] == "tinfoil-llama3-3-70b"
@pytest.mark.asyncio
async def test_model_mismatch_unknown_model_falls_back(self) -> None:
"""When the served model is not in the registry, use requested model."""
model_obj = MagicMock()
model_obj.id = "gpt-oss-120b"
model_obj.forwarded_model_id = "gpt-oss-120b"
with patch(
"routstr.proxy.get_model_instance",
return_value=None,
), patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=nonexistent",
model_obj,
100_000,
)
assert "actual_model" not in result
# calculate_cost called with the requested model (fallback)
call_args = mock_calc.call_args
assert call_args[0][0]["model"] == "gpt-oss-120b"
@pytest.mark.asyncio
async def test_old_format_no_model_uses_requested(self) -> None:
"""Old format without model field uses requested model for pricing."""
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=67,
output_tokens=42,
)
result = await _compute_ehbp_actual_cost(
"prompt=67,completion=42,total=109",
model_obj,
100_000,
)
assert "actual_model" not in result
call_args = mock_calc.call_args
assert call_args[0][0]["model"] == "llama3-3-70b"
@pytest.mark.asyncio
async def test_case_insensitive_model_match(self) -> None:
"""Casing differences between the header and forwarded_model_id
should not trigger a spurious mismatch."""
model_obj = MagicMock()
model_obj.id = "tinfoil-glm-5-2"
model_obj.forwarded_model_id = "glm-5-2" # lowercase
with patch(
"routstr.proxy.get_model_instance"
) as mock_get_model, patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
# Header returns uppercase — same model, different casing
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=GLM-5-2",
model_obj,
100_000,
)
assert "actual_model" not in result
# No mismatch: requested model pricing used
call_args = mock_calc.call_args
assert call_args[0][0]["model"] == "tinfoil-glm-5-2"
mock_get_model.assert_not_called()
@pytest.mark.asyncio
async def test_date_versioned_alias_resolves_to_requested(self) -> None:
"""When the served model is a date-versioned alias that resolves back
to the requested model, no mismatch is propagated."""
model_obj = MagicMock()
model_obj.id = "tinfoil-glm-5-2"
model_obj.forwarded_model_id = "glm-5-2"
resolved_model_obj = MagicMock()
resolved_model_obj.id = "other-provider-glm-5-2"
resolved_model_obj.forwarded_model_id = "glm-5-2"
with patch(
"routstr.proxy.get_model_instance",
return_value=resolved_model_obj,
) as mock_get_model, patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
# Tinfoil returns a date-versioned ID with different casing.
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=GLM-5-2-20260415",
model_obj,
100_000,
)
# Registry resolution, rather than unconditional suffix removal,
# establishes that this alias represents the expected model.
assert "actual_model" not in result
assert mock_calc.call_args[0][0]["model"] == "tinfoil-glm-5-2"
mock_get_model.assert_called_once_with("GLM-5-2-20260415")
@pytest.mark.asyncio
async def test_configured_date_version_is_preserved_as_identity(self) -> None:
"""A date suffix in forwarded_model_id is meaningful and preserved."""
model_obj = MagicMock()
model_obj.id = "tinfoil-glm-5-2-20260415"
model_obj.forwarded_model_id = "glm-5-2-20260415"
with patch(
"routstr.proxy.get_model_instance"
) as mock_get_model, patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=GLM-5-2-20260415",
model_obj,
100_000,
)
assert "actual_model" not in result
assert (
mock_calc.call_args[0][0]["model"]
== "tinfoil-glm-5-2-20260415"
)
mock_get_model.assert_not_called()
@pytest.mark.asyncio
async def test_different_client_alias_same_upstream_identity(self) -> None:
"""A global alias winner from another provider is not a failover when
its forwarded model ID matches the requested upstream identity."""
model_obj = MagicMock()
model_obj.id = "tinfoil-glm-5-2"
model_obj.forwarded_model_id = "glm-5-2"
resolved_model_obj = MagicMock()
resolved_model_obj.id = "other-provider-glm-5-2"
resolved_model_obj.forwarded_model_id = "GLM-5-2"
with patch(
"routstr.proxy.get_model_instance",
return_value=resolved_model_obj,
) as mock_get_model, patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=5,
output_msats=10,
total_msats=15,
total_usd=0.0,
input_tokens=42,
output_tokens=10,
)
result = await _compute_ehbp_actual_cost(
"prompt=42,completion=10,total=52,model=provider-alias",
model_obj,
100_000,
)
mock_get_model.assert_called_once_with("provider-alias")
assert "actual_model" not in result
assert mock_calc.call_args[0][0]["model"] == "tinfoil-glm-5-2"
# ---------------------------------------------------------------------------
# TinfoilUpstreamProvider
# ---------------------------------------------------------------------------
class TestTinfoilUpstreamProvider:
def test_provider_type_and_defaults(self) -> None:
assert TinfoilUpstreamProvider.provider_type == "tinfoil"
assert (
TinfoilUpstreamProvider.default_base_url
== "https://inference.tinfoil.sh"
)
assert TinfoilUpstreamProvider.supports_ehbp is True
def test_transform_model_name(self) -> None:
provider = TinfoilUpstreamProvider(api_key="test")
assert provider.transform_model_name("tinfoil/llama3-3-70b") == "llama3-3-70b"
assert provider.transform_model_name("llama3-3-70b") == "llama3-3-70b"
def test_get_ehbp_forwarding_target_includes_usage_header(self) -> None:
provider = TinfoilUpstreamProvider(api_key="test")
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
target = provider.get_ehbp_forwarding_target("v1/chat/completions", model_obj)
assert (
target.headers["X-Tinfoil-Request-Usage-Metrics"] == "true"
)
assert "v1/chat/completions" in target.url
def test_get_provider_metadata(self) -> None:
meta = TinfoilUpstreamProvider.get_provider_metadata()
assert meta["id"] == "tinfoil"
assert meta["name"] == "Tinfoil"
assert meta["fixed_base_url"] is True
def test_tinfoil_model_pricing_parses(self) -> None:
data = {
"id": "llama3-3-70b",
"context_window": 128000,
"created": 1721764788,
"pricing": {
"inputTokenPricePer1M": 1.75,
"outputTokenPricePer1M": 2.75,
"requestPrice": 0,
},
"endpoints": ["/v1/chat/completions"],
"type": "chat",
}
tf = TinfoilModel.parse_obj(data)
assert tf.id == "llama3-3-70b"
assert tf.pricing.inputTokenPricePer1M == 1.75
assert tf.pricing.outputTokenPricePer1M == 2.75
@pytest.mark.asyncio
async def test_fetch_models_parses_response(self) -> None:
provider = TinfoilUpstreamProvider(api_key="test")
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.raise_for_status = MagicMock()
mock_response.json.return_value = {
"data": [
{
"id": "llama3-3-70b",
"context_window": 128000,
"created": 1721764788,
"multimodal": False,
"pricing": {
"inputTokenPricePer1M": 1.75,
"outputTokenPricePer1M": 2.75,
"requestPrice": 0,
},
"endpoints": ["/v1/chat/completions"],
"type": "chat",
}
]
}
with patch("routstr.upstream.tinfoil.httpx.AsyncClient") as mock_client_cls:
mock_client = MagicMock()
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client.get = AsyncMock(return_value=mock_response)
mock_client_cls.return_value = mock_client
models = await provider.fetch_models()
assert len(models) == 1
assert models[0].id == "llama3-3-70b"
assert models[0].pricing.prompt == 1.75 / 1_000_000
assert models[0].pricing.completion == 2.75 / 1_000_000
assert models[0].context_length == 128000
@pytest.mark.asyncio
async def test_fetch_models_handles_error(self) -> None:
provider = TinfoilUpstreamProvider(api_key="test")
with patch("routstr.upstream.tinfoil.httpx.AsyncClient") as mock_client_cls:
mock_client = MagicMock()
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client.get = AsyncMock(side_effect=Exception("network error"))
mock_client_cls.return_value = mock_client
models = await provider.fetch_models()
assert models == []

View File

@@ -0,0 +1,90 @@
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock
import pytest
from routstr.upstream.tinfoil_trailer import forward_with_trailer
class FakeReader:
def __init__(self, chunks: list[bytes]) -> None:
self._chunks = chunks
async def read(self, _size: int) -> bytes:
if self._chunks:
return self._chunks.pop(0)
return b""
class FakeWriter:
def __init__(self) -> None:
self.written = b""
self.drain = AsyncMock()
self.wait_closed = AsyncMock()
self.close = MagicMock()
def write(self, data: bytes) -> None:
self.written += data
@pytest.mark.asyncio
async def test_forward_with_trailer_captures_usage_trailer(
monkeypatch: pytest.MonkeyPatch,
) -> None:
response = (
b"HTTP/1.1 200 OK\r\n"
b"Transfer-Encoding: chunked\r\n"
b"Trailer: X-Tinfoil-Usage-Metrics\r\n"
b"\r\n"
b"5\r\nhello\r\n"
b"0\r\n"
b"X-Tinfoil-Usage-Metrics: prompt=1,completion=2,total=3\r\n"
b"\r\n"
)
reader = FakeReader([response])
writer = FakeWriter()
open_connection = AsyncMock(return_value=(reader, writer))
monkeypatch.setattr(
"routstr.upstream.tinfoil_trailer.asyncio.open_connection", open_connection
)
result = await forward_with_trailer(
method="POST",
url="https://enclave.tinfoil.sh/v1/chat/completions?stream=true",
headers={"Authorization": "Bearer upstream"},
body=b"opaque",
)
assert result.status_code == 200
assert result.body == b"hello"
assert result.trailers == [
("x-tinfoil-usage-metrics", "prompt=1,completion=2,total=3")
]
assert b"Connection: close" in writer.written
writer.close.assert_called_once()
writer.wait_closed.assert_awaited_once()
@pytest.mark.asyncio
async def test_forward_with_trailer_enforces_response_size_limit(
monkeypatch: pytest.MonkeyPatch,
) -> None:
response = b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\nhello"
reader = FakeReader([response])
writer = FakeWriter()
monkeypatch.setattr(
"routstr.upstream.tinfoil_trailer.asyncio.open_connection",
AsyncMock(return_value=(reader, writer)),
)
with pytest.raises(ValueError, match="EHBP response exceeded"):
await forward_with_trailer(
method="POST",
url="https://enclave.tinfoil.sh/v1/chat/completions",
headers={},
body=b"opaque",
max_response_bytes=4,
)
writer.close.assert_called_once()

2
uv.lock generated
View File

@@ -2443,6 +2443,7 @@ dependencies = [
{ name = "cashu" },
{ name = "fastapi", extra = ["standard"] },
{ name = "greenlet" },
{ name = "h11" },
{ name = "httpx", extra = ["socks"] },
{ name = "litellm" },
{ name = "marshmallow" },
@@ -2477,6 +2478,7 @@ requires-dist = [
{ name = "cashu", specifier = ">=0.20" },
{ name = "fastapi", extras = ["standard"], specifier = ">=0.115" },
{ name = "greenlet", specifier = ">=3.2.1" },
{ name = "h11", specifier = ">=0.14" },
{ name = "httpx", extras = ["socks"], specifier = ">=0.25.2" },
{ name = "litellm", specifier = ">=1.55.0" },
{ name = "marshmallow", specifier = ">=3.13,<4.0" },