laantungir/routstr-core
1
0
Fork 0
mirror of https://github.com/Routstr/routstr-core.git synced 2026-07-22 12:22:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add Tinfoil direct blind-upstream integration

Browse Source 
Add TinfoilUpstreamProvider that uses inference.tinfoil.sh as a direct
EHBP upstream. Routstr acts as a blind relay: it forwards the opaque
encrypted body to the Tinfoil enclave without ever seeing plaintext,
and bills from the X-Tinfoil-Usage-Metrics response header.

- New routstr/upstream/tinfoil.py: fetches models from public
  GET /v1/models, parses Tinfoil pricing into standard Model/Pricing
  schema, supports_ehbp=True, proxies /attestation to atc.tinfoil.sh
- Updated routstr/upstream/ehbp.py:
  - parse_tinfoil_usage_metrics() parses prompt=N,completion=N header
  - _resolve_ehbp_target_url() honors X-Tinfoil-Enclave-Url from SDK
  - _strip_proxy_headers() removes proxy-only headers before forwarding
  - _compute_ehbp_actual_cost() converts usage to msats via calculate_cost
  - forward_ehbp_request() finalizes with exact token cost when usage
    header is present (non-streaming), falls back to max-cost otherwise
  - forward_ehbp_x_cashu_request() computes refund from actual cost
    when usage is available
- Updated routstr/proxy.py: forward /attestation and /.well-known/ paths
  without model/cost/auth lookups
- Updated routstr/upstream/__init__.py and helpers.py: register and
  auto-seed Tinfoil provider from TINFOIL_API_KEY env var
- Updated .env.example with TINFOIL_API_KEY
- 22 new unit tests in tests/unit/test_tinfoil_integration.py
- Updated docs/tinfoil-direct-integration.md and docs/ehbp-proxy-support.md
  with implementation status and billing behavior table
This commit is contained in:
redshift
2026-06-21 13:16:01 +08:00
parent 24b90af6e6
commit 2e350e082b
9 changed files with 1178 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.env.example
View File

@@ -2,6 +2,9 @@
UPSTREAM_BASE_URL=https://api.openai.com/v1
UPSTREAM_API_KEY=your-upstream-api-key
# Tinfoil (confidential inference enclaves, EHBP)
# TINFOIL_API_KEY=your-tinfoil-api-key
# ADMIN_PASSWORD=secure-admin-password
# Database

14
docs/ehbp-proxy-support.md
View File

@@ -128,6 +128,20 @@ Three parties see three different model IDs:
| PPQ.AI billing | `X-Private-Model` header | `private/kimi-k2-6` | Proxy sends `forwarded_model_id` |
| Tinfoil enclave | `body.model` (encrypted) | `kimi-k2-6` | SDK strips `tinfoil-` prefix before encryption |
## Implementation status
A dedicated `TinfoilUpstreamProvider` (`routstr/upstream/tinfoil.py`) now
implements the direct blind-upstream pattern described above. The shared EHBP
helpers in `routstr/upstream/ehbp.py` were extended to:
- Request usage metrics via `X-Tinfoil-Request-Usage-Metrics: true`.
- Parse `X-Tinfoil-Usage-Metrics` from the response header (non-streaming).
- Override the forwarding URL with `X-Tinfoil-Enclave-Url` when the SDK sends it.
- Finalize bearer billing with actual token cost via `adjust_payment_for_tokens`.
- Compute X-Cashu refunds from actual cost instead of max cost.
See `docs/tinfoil-direct-integration.md` for the full implementation notes.
## Not yet tested
These changes were written without integration testing due to the complexity

442
docs/tinfoil-direct-integration.md Normal file
View File

@@ -0,0 +1,442 @@
# Tinfoil / PPQ Private-Mode Integration Notes
This document summarizes the current options for integrating Tinfoil/PPQ private models with Routstr, based on the EHBP work in this branch and local testing against `ppq-private-mode-proxy`.
## Background
PPQ private models run behind a Tinfoil/EHBP flow:
- Request bodies are HPKE-encrypted by a Tinfoil client.
- The PPQ `/private/` endpoint routes ciphertext to the attested enclave.
- The enclave decrypts, runs inference, and returns an encrypted response.
- The caller's Tinfoil client decrypts the response locally.
The PPQ private-mode proxy (`~/projects/ppq-private-mode-proxy`) uses this pattern with the JavaScript `tinfoil` SDK:
```ts
import { SecureClient } from "tinfoil";
const apiBase = "https://api.ppq.ai";
const client = new SecureClient({
baseURL: `${apiBase}/private/`,
attestationBundleURL: `${apiBase}/private`,
transport: "ehbp",
});
await client.ready();
const response = await client.fetch(`${apiBase}/private/v1/chat/completions`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.PPQ_API_KEY}`,
"X-Private-Model": "private/gpt-oss-120b",
"x-query-source": "api",
},
body: JSON.stringify({
model: "gpt-oss-120b", // enclave-internal model id
messages: [{ role: "user", content: "Hello" }],
}),
});
const json = await response.json();
console.log(json.usage);
```
`X-Private-Model` carries the PPQ-facing private model id, while the encrypted JSON body uses the enclave-internal model id without the `private/` prefix.
## PPQ private model pricing
PPQ exposes private model pricing through:
```text
GET https://api.ppq.ai/v1/models?type=all
```
Filter models whose IDs start with `private/`.
Example private pricing observed:
| Model | Input USD / 1M tokens | Output USD / 1M tokens |
|---|---:|---:|
| `private/gpt-oss-120b` | `0.79125` | `1.31875` |
| `private/llama3-3-70b` | `1.84625` | `2.90125` |
| `private/qwen3-vl-30b` | `1.31875` | `4.22` |
| `private/glm-5-2` | `1.5825` | `5.53875` |
| `private/gemma4-31b` | `0.47475` | `1.055` |
| `private/kimi-k2-6` | `1.5825` | `5.53875` |
The `ppq-private-mode-proxy` commit `ba984214793d3bca0f7d046b6955d42abc1c6843` changed OpenClaw display metadata to align with a 5% API margin. The live PPQ model endpoint returns the more precise rates above, which already include that margin.
Actual PPQ private billing is:
```text
price_usd =
input_tokens * input_per_1M_tokens / 1_000_000
+ output_tokens * output_per_1M_tokens / 1_000_000
```
A test request to `private/gpt-oss-120b` produced:
```json
{
"input_count": 74,
"output_count": 3,
"price_in_usd": 0.00006250875
}
```
which exactly matches:
```text
74 * 0.79125 / 1_000_000 + 3 * 1.31875 / 1_000_000
= 0.00006250875 USD
```
## Integration architectures
There are three materially different ways Routstr could integrate Tinfoil/PPQ private inference.
## Option A: User integrates Tinfoil directly
```text
User app / SDK
-> Tinfoil SecureClient / TinfoilAI
-> EHBP-encrypted request
-> Tinfoil/PPQ private enclave
```
Properties:
- Best privacy for the user.
- Routstr is not in the request path.
- User's client encrypts requests and decrypts responses.
- Usage is visible to the user's app after decryption.
- Billing is handled directly by Tinfoil/PPQ.
Example with Tinfoil's OpenAI-compatible client:
```ts
import { TinfoilAI } from "tinfoil";
const client = new TinfoilAI({
apiKey: process.env.TINFOIL_API_KEY,
transport: "ehbp",
});
const res = await client.chat.completions.create({
model: "llama3-3-70b",
messages: [{ role: "user", content: "Hello" }],
});
console.log(res.choices[0].message.content);
console.log(res.usage);
```
This is not a Routstr marketplace flow unless Routstr only acts as discovery/UI around direct Tinfoil/PPQ usage.
## Option B: Routstr integrates Tinfoil as an upstream client
```text
User -> Routstr plaintext request
-> Routstr Tinfoil SecureClient encrypts to PPQ/Tinfoil
-> PPQ private enclave
-> Routstr receives decrypted response
-> Routstr bills from decrypted usage
-> Routstr returns plaintext response to user
```
Properties:
- Easier exact billing.
- Routstr can read the decrypted OpenAI response and `usage` object.
- Routstr can charge exact PPQ token pricing.
- Privacy is different: the user sends plaintext to Routstr, and Routstr sees prompts/responses.
- End-to-end encryption is only Routstr-to-enclave, not user-to-enclave.
This should be considered a separate product/provider mode, not the same as an end-to-end private relay.
Practical implementation options:
1. Run a Node sidecar that uses the `tinfoil` npm package and expose it as a local HTTP upstream to Routstr.
2. Port EHBP client behavior to Python.
3. Reuse or adapt `ppq-private-mode-proxy` as a local upstream.
A Node sidecar is probably the quickest implementation path because `ppq-private-mode-proxy` already demonstrates the full flow.
## Option C: Routstr uses Tinfoil as a direct blind upstream
This is the current branch's design intent and is likely the best fit if Tinfoil/PPQ exposes usage metadata headers:
```text
User Tinfoil SecureClient
-> encrypted request body
-> Routstr proxy
-> Tinfoil/PPQ private API
with X-Tinfoil-Request-Usage-Metrics: true
-> PPQ private enclave
<- encrypted response
plus X-Tinfoil-Usage-Metrics / cost headers
<- Routstr proxy
-> user decrypts response
```
In this mode Routstr is still a normal upstream proxy from the user's point of view, but the upstream is Tinfoil/PPQ private inference and the body remains opaque to Routstr.
Properties:
- Strongest privacy with Routstr in the path.
- Routstr never sees plaintext prompt or plaintext response.
- Routstr can authenticate and route based on plaintext headers.
- Routstr should request Tinfoil usage metadata by adding `X-Tinfoil-Request-Usage-Metrics: true` to the upstream request.
- Tinfoil documents `X-Tinfoil-Usage-Metrics` as an upstream response header for non-streaming requests.
- For streaming requests, Tinfoil documents `X-Tinfoil-Usage-Metrics` as an HTTP trailer available only after the response body completes.
- If Tinfoil/PPQ returns `X-Tinfoil-Usage-Metrics` or a cost header on the encrypted response, Routstr can bill exactly without decrypting the body.
- If usage is only present inside the encrypted response body, Routstr still cannot read it and exact billing is not possible without a separate metadata path.
This is the only architecture that preserves end-to-end encryption from the user to the PPQ/Tinfoil enclave while still letting Routstr mediate payment. The key requirement is that usage/cost metadata must be returned outside the encrypted body, ideally as a response header available before body streaming begins.
## Current Routstr problem
The current EHBP implementation charges successful EHBP requests at `max_cost_for_model` because Routstr cannot decrypt the response body:
```text
successful EHBP request -> charge full reserved max cost
```
That is incorrect for PPQ private models. Max cost should be only a reservation/solvency ceiling. Final charge should use actual PPQ private token pricing.
Desired behavior:
```text
reserve max cost
forward encrypted request
obtain actual usage/cost metadata
finalize actual cost
refund/release the difference
```
## Usage/cost metadata requirement
For blind-relay exact billing, PPQ should return one of the following outside the encrypted response body:
```http
X-PPQ-Cost-USD: 0.00006250875
```
or:
```http
X-Private-Usage-Metrics: input=74,output=3
```
or:
```http
X-Tinfoil-Usage-Metrics: prompt=74,completion=3,total=77
```
Tinfoil proxy documentation references a usage-metrics flow where a proxy can request usage via:
```http
X-Tinfoil-Request-Usage-Metrics: true
```
and read usage from:
```http
X-Tinfoil-Usage-Metrics
```
Docs/example references:
- https://docs.tinfoil.sh/guides/proxy-server
- https://github.com/tinfoilsh/encrypted-request-proxy-example
During local PPQ testing, PPQ responses included this CORS exposure header:
```http
Access-Control-Expose-Headers: Ehbp-Response-Nonce, X-Private-Usage-Metrics, X-Encrypted-Usage-Metrics, X-Tinfoil-Usage-Metrics
```
However, the actual tested non-streaming response did not include any of these usage headers, even when `X-Tinfoil-Request-Usage-Metrics: true` was sent.
The decrypted body did include normal OpenAI usage, but only the decrypting Tinfoil client can see that body.
## Query-history fallback
PPQ's query history endpoint exposes actual usage and cost:
```text
GET https://api.ppq.ai/queries/history?page=1&page_count=...
```
A record includes:
```json
{
"timestamp": "...",
"model": "private/gpt-oss-120b",
"input_count": 74,
"output_count": 3,
"price_in_usd": 0.00006250875,
"query_type": "chat_completion",
"query_source": "api"
}
```
This could be used as a fallback, but it is less robust than response headers/trailers because matching a request to a history row can be race-prone under concurrency. It would need a reliable request identifier or metadata field that PPQ stores in history.
## Recommended Routstr direction
Use Tinfoil/PPQ as a direct blind upstream and have Routstr explicitly request usage metadata:
```text
User encrypts body
Routstr reserves max cost
Routstr forwards encrypted body to Tinfoil/PPQ /private/
Routstr includes X-Tinfoil-Request-Usage-Metrics: true
Tinfoil/PPQ returns X-Tinfoil-Usage-Metrics as a response header for non-streaming,
or as an HTTP trailer after the body completes for streaming
Routstr finalizes exact charge
User decrypts encrypted response
```
This preserves both:
- privacy: Routstr cannot read prompts/responses;
- exact billing: Routstr can charge actual PPQ private model cost, assuming Tinfoil/PPQ returns usage/cost metadata outside the encrypted body.
Implementation steps:
1. Update PPQ model fetching to include private models:
```text
GET https://api.ppq.ai/v1/models?type=all
```
2. Register `private/*` models and any Routstr-facing aliases with correct `forwarded_model_id`.
3. Keep max-cost reservation for bearer keys and X-Cashu solvency checks.
4. Add parsing support for possible usage/cost headers:
```http
X-PPQ-Cost-USD
X-Private-Usage-Metrics
X-Tinfoil-Usage-Metrics
X-Encrypted-Usage-Metrics
```
5. Finalize by actual cost instead of max cost:
```text
actual_msats = ceil((actual_usd / sats_usd_price()) * 1000)
actual_msats = max(actual_msats, settings.min_request_msat)
actual_msats = min(actual_msats, reserved_msats)
```
or, if only token counts are available:
```text
actual_usd =
input_tokens * input_per_1M_tokens / 1_000_000
+ output_tokens * output_per_1M_tokens / 1_000_000
```
6. If usage/cost metadata is missing, choose an explicit policy:
- fail closed and refund/revert;
- query PPQ history as a fallback;
- fallback to max-cost billing only if explicitly configured and clearly disclosed.
Silent max-cost billing should not be the default for PPQ private requests.
## X-Cashu consideration
For bearer-auth requests, finalization can happen after the response stream completes if usage is delivered as a trailer.
For `X-Cashu`, Routstr needs to return the refund token in the response headers. If usage/cost is only available after consuming the encrypted response stream, Routstr may need to buffer EHBP responses before sending them to the client so it can compute the refund amount first.
Possible approaches:
1. Prefer a non-trailer response header with actual cost, available before streaming body starts.
2. Buffer EHBP X-Cashu responses and then return `X-Cashu` refund.
3. Introduce a later/refund-claim mechanism, which would be a larger protocol change.
## Summary
- PPQ private models are billed per actual input/output tokens.
- Private model rates are available from `GET /v1/models?type=all`.
- Current Routstr EHBP billing at max cost is wrong for PPQ private models.
- Direct Tinfoil integration inside Routstr would enable exact usage billing but would make Routstr see plaintext.
- A blind EHBP relay preserves privacy but requires PPQ/Tinfoil to expose usage/cost in plaintext headers/trailers.
- The preferred solution is to keep Routstr blind and have PPQ return billing metadata outside the encrypted body.
## Implementation status
Direct Tinfoil upstream integration is implemented in `routstr/upstream/tinfoil.py`
and `routstr/upstream/ehbp.py`.
### What was built
- `TinfoilUpstreamProvider` (`provider_type = "tinfoil"`):
- Base URL: `https://inference.tinfoil.sh`
- Fetches models from the public `GET /v1/models` endpoint (no auth needed).
- Parses Tinfoil's pricing (`inputTokenPricePer1M`, `outputTokenPricePer1M`,
`requestPrice`) into the standard `Model`/`Pricing` schema.
- `supports_ehbp = True` — acts as a blind EHBP relay.
- `get_ehbp_forwarding_target()` returns a target that includes
`X-Tinfoil-Request-Usage-Metrics: true`.
- `forward_get_request()` proxies `/attestation` to `https://atc.tinfoil.sh/attestation`
so the SDK can fetch attestation bundles through Routstr.
- Registered in `routstr/upstream/__init__.py` and seeded from
`TINFOIL_API_KEY` env var.
- `routstr/upstream/ehbp.py`:
- `parse_tinfoil_usage_metrics()` parses `prompt=N,completion=N[,total=N]`
into an OpenAI-style usage dict.
- `_resolve_ehbp_target_url()` overrides the forwarding URL with
`X-Tinfoil-Enclave-Url` when the SDK sends it.
- `_strip_proxy_headers()` removes `X-Routstr-Model`,
`X-Tinfoil-Enclave-Url`, and `X-Tinfoil-Request-Usage-Metrics` before
forwarding to the enclave.
- `_compute_ehbp_actual_cost()` converts the usage header into msats via
`calculate_cost()`, clamped to `[min_request_msat, max_cost_for_model]`.
- `forward_ehbp_request()` (bearer auth): if `X-Tinfoil-Usage-Metrics` is
present in the response header, finalizes with `adjust_payment_for_tokens()`
for exact billing; otherwise falls back to max-cost.
- `forward_ehbp_x_cashu_request()`: if usage is available, computes the
refund from actual cost instead of max cost.
- `routstr/proxy.py`: `/attestation` and `/.well-known/` paths are forwarded
to all enabled upstreams without model/cost/auth lookups.
### Billing behavior
| Request shape | Usage source | Billing |
|---|---|---|
| Bearer, non-streaming | `X-Tinfoil-Usage-Metrics` response header | Exact token cost via `adjust_payment_for_tokens` |
| Bearer, streaming | HTTP trailer (not available before body) | Max-cost fallback |
| X-Cashu, non-streaming | `X-Tinfoil-Usage-Metrics` response header | Refund = `redeemed - actual_cost` |
| X-Cashu, streaming | HTTP trailer | Refund = `redeemed - max_cost` |
### Setup
```bash
TINFOIL_API_KEY=your-tinfoil-api-key
```
The provider is auto-seeded on first startup.
### What still needs verification
- End-to-end test with a real Tinfoil SDK client against a Routstr node with
`TINFOIL_API_KEY` set.
- Streaming requests: usage is delivered as an HTTP trailer. Currently the
bearer path finalizes max-cost before streaming begins. Supporting streaming
usage would require buffering the response (for X-Cashu) or a deferred
finalization (for bearer).
- Whether Tinfoil's `/v1/responses` endpoint also returns usage metrics
headers.

11
routstr/proxy.py
View File

@@ -166,6 +166,8 @@ _API_PATH_PREFIXES = (
"moderations",
"providers",
"tee/",
"attestation",
".well-known/",
)
@@ -208,9 +210,12 @@ async def proxy(
else:
model_id = request_body_dict.get("model", "unknown")
# /tee/* GET requests (e.g. attestation) don't map to models — just
# forward to all enabled upstreams without model/cost/auth lookups.
if request.method == "GET" and path.startswith("tee/"):
# /tee/* and /attestation GET requests (e.g. Tinfoil attestation bundle)
# don't map to models — just forward to all enabled upstreams without
# model/cost/auth lookups.
if request.method == "GET" and (
path.startswith("tee/") or path.startswith("attestation")
):
all_upstreams = _upstreams
last_error_response = None
for i, upstream in enumerate(all_upstreams):

2
routstr/upstream/__init__.py
View File

@@ -11,6 +11,7 @@ from .openrouter import OpenRouterUpstreamProvider
from .perplexity import PerplexityUpstreamProvider
from .ppqai import PPQAIUpstreamProvider
from .routstr import RoutstrUpstreamProvider
from .tinfoil import TinfoilUpstreamProvider
from .xai import XAIUpstreamProvider
upstream_provider_classes: list[type[BaseUpstreamProvider]] = [
@@ -26,6 +27,7 @@ upstream_provider_classes: list[type[BaseUpstreamProvider]] = [
PerplexityUpstreamProvider,
PPQAIUpstreamProvider,
RoutstrUpstreamProvider,
TinfoilUpstreamProvider,
XAIUpstreamProvider,
]
"""List of all upstream classes"""

201
routstr/upstream/ehbp.py
View File

@@ -13,7 +13,12 @@ from fastapi.responses import Response, StreamingResponse
from sqlalchemy import case
from sqlmodel import col, update
from ..auth import ROUTSTR_FEE_PERCENT, get_billing_key, payments_logger
from ..auth import (
ROUTSTR_FEE_PERCENT,
adjust_payment_for_tokens,
get_billing_key,
payments_logger,
)
from ..core import get_logger
from ..core.db import (
ApiKey,
@@ -22,12 +27,139 @@ from ..core.db import (
store_cashu_transaction,
)
from ..core.exceptions import UpstreamError
from ..core.settings import settings
from ..payment.cost_calculation import (
CostData,
MaxCostData,
calculate_cost,
)
from ..payment.helpers import create_error_response
from ..payment.models import Model
from ..wallet import recieve_token, send_token
logger = get_logger(__name__)
# Headers that the Tinfoil SDK sends to tell the proxy where to forward the
# encrypted request, and the request/response usage-metrics pair.
_ENCLAVE_URL_HEADER = "X-Tinfoil-Enclave-Url"
_REQUEST_USAGE_HEADER = "X-Tinfoil-Request-Usage-Metrics"
_RESPONSE_USAGE_HEADER = "X-Tinfoil-Usage-Metrics"
# Headers that must not be forwarded to the upstream enclave.
_PROXY_ONLY_HEADERS = {
"x-routstr-model",
"x-tinfoil-enclave-url",
"x-tinfoil-request-usage-metrics",
}
def parse_tinfoil_usage_metrics(header_value: str | None) -> dict | None:
"""Parse ``X-Tinfoil-Usage-Metrics`` into an OpenAI-style usage dict.
The header format is ``prompt=<n>,completion=<n>,total=<n>``. Returns a dict
like ``{"prompt_tokens": n, "completion_tokens": n}`` suitable for
:func:`calculate_cost`, or ``None`` when the header is absent or malformed.
"""
if not header_value:
return None
parts: dict[str, int] = {}
for item in header_value.split(","):
key, sep, value = item.partition("=")
if not sep:
continue
try:
parts[key.strip()] = int(value.strip())
except (ValueError, TypeError):
continue
prompt = parts.get("prompt")
completion = parts.get("completion")
if prompt is not None and completion is not None:
result: dict[str, int] = {
"prompt_tokens": prompt,
"completion_tokens": completion,
}
if "total" in parts:
result["total_tokens"] = parts["total"]
return result
return None
def _resolve_ehbp_target_url(
target_url: str, path: str, headers: Mapping[str, str]
) -> str:
"""Override the forwarding URL with ``X-Tinfoil-Enclave-Url`` if present.
When the Tinfoil SDK is configured with a proxy ``baseURL``, it sends the
actual enclave URL in ``X-Tinfoil-Enclave-Url``. The proxy must forward to
that URL, not to its own default, so the encrypted payload reaches the same
enclave the client verified.
"""
enclave_url = (
headers.get(_ENCLAVE_URL_HEADER)
or headers.get(_ENCLAVE_URL_HEADER.lower())
or headers.get(_ENCLAVE_URL_HEADER.upper())
)
if enclave_url:
return f"{enclave_url.rstrip('/')}/{path.lstrip('/')}"
return target_url
def _strip_proxy_headers(headers: dict[str, str]) -> dict[str, str]:
"""Remove proxy-routing headers that must not reach the upstream enclave."""
clean = {}
for key, value in headers.items():
if key.lower() not in _PROXY_ONLY_HEADERS:
clean[key] = value
return clean
async def _compute_ehbp_actual_cost(
usage_header: str | None,
model_obj: Model,
max_cost_for_model: int,
) -> int:
"""Compute the actual cost in msats from Tinfoil usage metrics.
Falls back to ``max_cost_for_model`` when usage is absent (streaming) or
cannot be priced. The result is clamped to ``[min_request_msat,
max_cost_for_model]`` so the refund never exceeds the reservation and is
never zero.
"""
usage_dict = parse_tinfoil_usage_metrics(usage_header)
if usage_dict is None:
return max_cost_for_model
try:
cost = await calculate_cost(
{"model": model_obj.id, "usage": usage_dict},
max_cost_for_model,
)
except Exception as e:
logger.warning(
"EHBP usage cost calculation failed, falling back to max cost",
extra={
"model": model_obj.id,
"error": str(e),
"usage": usage_dict,
},
)
return max_cost_for_model
if isinstance(cost, MaxCostData):
return max_cost_for_model
if isinstance(cost, CostData):
actual = max(int(cost.total_msats), int(settings.min_request_msat))
return min(actual, max_cost_for_model)
# CostDataError
logger.warning(
"EHBP usage cost calculation error, falling back to max cost",
extra={
"model": model_obj.id,
"error": getattr(cost, "message", str(cost)),
},
)
return max_cost_for_model
@dataclass(frozen=True)
class EHBPForwardingTarget:
@@ -193,17 +325,24 @@ async def forward_ehbp_request(
session: AsyncSession,
model_obj: Model,
) -> Response | StreamingResponse:
"""Forward an EHBP bearer-auth request and finalize max-cost billing."""
"""Forward an EHBP bearer-auth request and finalize billing.
Sends ``X-Tinfoil-Request-Usage-Metrics: true`` so the enclave returns token
counts in the ``X-Tinfoil-Usage-Metrics`` response header (non-streaming) or
trailer (streaming). When usage is available in the response header,
billing is finalized to the actual token cost via
:func:`adjust_payment_for_tokens`. When usage is not available (streaming
or unsupported upstream), billing falls back to max-cost.
"""
target = upstream.get_ehbp_forwarding_target(path, model_obj) # type: ignore[attr-defined]
upstream_headers = {**headers, **dict(target.headers)}
upstream_headers.pop("x-routstr-model", None)
upstream_headers.pop("X-Routstr-Model", None)
target_url = _resolve_ehbp_target_url(target.url, path, headers)
upstream_headers = _strip_proxy_headers({**headers, **dict(target.headers)})
provider_type = getattr(upstream, "provider_type", "unknown")
logger.debug(
"Forwarding EHBP request to upstream",
extra={
"url": target.url,
"url": target_url,
"method": request.method,
"path": path,
"model": model_obj.id,
@@ -221,7 +360,7 @@ async def forward_ehbp_request(
response = await client.send(
client.build_request(
request.method,
target.url,
target_url,
headers=upstream_headers,
content=request_body,
params=upstream.prepare_params(path, request.query_params), # type: ignore[attr-defined]
@@ -255,6 +394,29 @@ async def forward_ehbp_request(
status_code=response.status_code,
)
# Check for usage metrics in the response header (non-streaming case).
# For streaming requests, usage is delivered as an HTTP trailer after
# the body completes and is not available here — fall back to max-cost.
usage_header = response.headers.get(_RESPONSE_USAGE_HEADER)
usage_dict = parse_tinfoil_usage_metrics(usage_header)
if usage_dict is not None:
logger.info(
"EHBP usage metrics received, finalizing with actual token cost",
extra={
"model": model_obj.id,
"provider": provider_type,
"usage": usage_dict,
"key_hash": key.hashed_key[:8] + "...",
},
)
await adjust_payment_for_tokens(
key,
{"model": model_obj.id, "usage": usage_dict},
session,
max_cost_for_model,
)
else:
await finalize_ehbp_max_cost_payment(
key, session, max_cost_for_model, model_obj.id
)
@@ -306,9 +468,11 @@ async def forward_ehbp_x_cashu_request(
) -> Response | StreamingResponse:
"""Redeem X-Cashu, forward EHBP opaquely, and refund unspent value.
Since the response is encrypted, usage cannot be inspected. Successful EHBP
X-Cashu requests are charged at max_cost_for_model and any excess token
value is refunded.
When the upstream returns ``X-Tinfoil-Usage-Metrics`` in the response
header (non-streaming), the refund is computed from the actual token cost
instead of max_cost_for_model. For streaming requests, usage is only
available as an HTTP trailer after the body completes and the refund
falls back to max_cost_for_model.
"""
request_id = getattr(request.state, "request_id", None)
amount = 0
@@ -334,9 +498,8 @@ async def forward_ehbp_x_cashu_request(
headers = upstream.prepare_headers(dict(request.headers)) # type: ignore[attr-defined]
target = upstream.get_ehbp_forwarding_target(path, model_obj) # type: ignore[attr-defined]
upstream_headers = {**headers, **dict(target.headers)}
upstream_headers.pop("x-routstr-model", None)
upstream_headers.pop("X-Routstr-Model", None)
target_url = _resolve_ehbp_target_url(target.url, path, headers)
upstream_headers = _strip_proxy_headers({**headers, **dict(target.headers)})
request_body = await request.body()
client = httpx.AsyncClient(
@@ -348,7 +511,7 @@ async def forward_ehbp_x_cashu_request(
response = await client.send(
client.build_request(
request.method,
target.url,
target_url,
headers=upstream_headers,
content=request_body,
params=upstream.prepare_params(path, request.query_params), # type: ignore[attr-defined]
@@ -377,8 +540,14 @@ async def forward_ehbp_x_cashu_request(
error_response.headers["X-Cashu"] = refund_token
return error_response
refund_amount = amount - _msats_to_unit_amount(max_cost_for_model, unit)
response_headers = dict(response.headers)
# Compute refund from actual usage when available (non-streaming),
# otherwise fall back to max_cost_for_model.
usage_header = response.headers.get(_RESPONSE_USAGE_HEADER)
actual_cost_msats = await _compute_ehbp_actual_cost(
usage_header, model_obj, max_cost_for_model
)
refund_amount = amount - _msats_to_unit_amount(actual_cost_msats, unit)
response_headers = _strip_proxy_headers(dict(response.headers))
if refund_amount > 0:
response_headers["X-Cashu"] = await send_cashu_refund(
refund_amount, unit, mint, request_id

1
routstr/upstream/helpers.py
View File

@@ -264,6 +264,7 @@ async def _seed_providers_from_settings(
("PERPLEXITY_API_KEY", "perplexity", None, None),
("FIREWORKS_API_KEY", "fireworks", None, None),
("XAI_API_KEY", "xai", None, None),
("TINFOIL_API_KEY", "tinfoil", None, None),
]
for env_key, provider_type, _, _ in env_mappings:

218
routstr/upstream/tinfoil.py Normal file
View File

@@ -0,0 +1,218 @@
from __future__ import annotations
from typing import TYPE_CHECKING
import httpx
from fastapi import Request
from fastapi.responses import Response, StreamingResponse
from pydantic.v1 import BaseModel, Field
from ..core.exceptions import UpstreamError
from ..core.logging import get_logger
from ..payment.models import Architecture, Model, Pricing
from .base import BaseUpstreamProvider
from .ehbp import EHBPForwardingTarget
if TYPE_CHECKING:
from ..core.db import UpstreamProviderRow
logger = get_logger(__name__)
class TinfoilModelPricing(BaseModel):
inputTokenPricePer1M: float = Field(0.0)
outputTokenPricePer1M: float = Field(0.0)
requestPrice: float = 0.0
class TinfoilModel(BaseModel):
id: str
context_window: int = 0
created: int = 0
multimodal: bool = False
reasoning: bool = False
tool_calling: bool = False
type: str = "chat"
pricing: TinfoilModelPricing = TinfoilModelPricing()
endpoints: list[str] = []
class TinfoilUpstreamProvider(BaseUpstreamProvider):
"""Direct upstream provider for the Tinfoil inference API.
Tinfoil hosts open-source models inside attested secure enclaves and exposes
an OpenAI-compatible API at ``https://inference.tinfoil.sh``. Request and
response bodies are encrypted end-to-end with EHBP (HPKE), so Routstr acts
as a blind relay: it forwards the opaque encrypted body, never sees
plaintext, and bills from the ``X-Tinfoil-Usage-Metrics`` header that
Tinfoil returns outside the encrypted body when
``X-Tinfoil-Request-Usage-Metrics: true`` is set.
"""
provider_type = "tinfoil"
default_base_url = "https://inference.tinfoil.sh"
platform_url = "https://docs.tinfoil.sh"
supports_ehbp = True
def __init__(self, api_key: str, provider_fee: float = 1.0):
super().__init__(
base_url=self.default_base_url,
api_key=api_key,
provider_fee=provider_fee,
)
@classmethod
def from_db_row(
cls, provider_row: "UpstreamProviderRow"
) -> "TinfoilUpstreamProvider":
return cls(
api_key=provider_row.api_key,
provider_fee=provider_row.provider_fee,
)
@classmethod
def get_provider_metadata(cls) -> dict[str, object]:
return {
"id": cls.provider_type,
"name": "Tinfoil",
"default_base_url": cls.default_base_url,
"fixed_base_url": True,
"platform_url": cls.platform_url,
"can_create_account": False,
"can_topup": False,
"can_show_balance": False,
}
def transform_model_name(self, model_id: str) -> str:
return model_id.removeprefix("tinfoil/")
async def forward_get_request(
self,
request: Request,
path: str,
headers: dict,
) -> Response | StreamingResponse:
"""Handle Tinfoil-specific GET endpoints.
* ``/attestation`` (or ``/tee/attestation``): proxy to the Tinfoil ATC
(attestation bundle proxy) at ``https://atc.tinfoil.sh/attestation``.
* Other GETs: forward to the enclave URL from ``X-Tinfoil-Enclave-Url``
when present, otherwise to the provider base URL.
"""
clean_path = path.removeprefix("tee/")
if clean_path == "attestation":
return await self._proxy_attestation(headers)
return await super().forward_get_request(request, path, headers)
async def _proxy_attestation(self, headers: dict) -> Response:
url = "https://atc.tinfoil.sh/attestation"
async with httpx.AsyncClient(
transport=httpx.AsyncHTTPTransport(retries=1),
timeout=30.0,
) as client:
try:
resp = await client.get(
url,
headers={
"Accept": headers.get("accept", "application/json"),
},
)
response_headers = dict(resp.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
return Response(
content=resp.content,
status_code=resp.status_code,
headers=response_headers,
)
except Exception as exc:
raise UpstreamError(
f"Error fetching Tinfoil attestation: {type(exc).__name__}",
status_code=502,
) from exc
def get_ehbp_forwarding_target(
self, path: str, model_obj: Model
) -> EHBPForwardingTarget:
"""Return the Tinfoil enclave target for EHBP requests.
Requests usage metrics from the enclave so Routstr can bill exactly
without decrypting the response body. The actual forwarding URL is
overridden at dispatch time by ``X-Tinfoil-Enclave-Url`` when the SDK
sends it (see ``routstr/upstream/ehbp.py``).
"""
return EHBPForwardingTarget(
url=f"{self.base_url.rstrip('/')}/{path.lstrip('/')}",
headers={"X-Tinfoil-Request-Usage-Metrics": "true"},
)
async def fetch_models(self) -> list[Model]:
"""Fetch models from the public Tinfoil models endpoint.
``GET /v1/models`` is unauthenticated and returns all available models
with their pricing in USD per 1M tokens.
"""
url = f"{self.base_url}/v1/models"
try:
async with httpx.AsyncClient(timeout=30.0) as client:
response = await client.get(url)
response.raise_for_status()
data = response.json()
models_data = data.get("data", [])
models: list[Model] = []
for model_data in models_data:
try:
tf = TinfoilModel.parse_obj(model_data)
input_price = tf.pricing.inputTokenPricePer1M
output_price = tf.pricing.outputTokenPricePer1M
request_price = tf.pricing.requestPrice
modality = "text->text"
input_modalities = ["text"]
output_modalities = ["text"]
if tf.multimodal:
modality = "text->text+image"
input_modalities = ["text", "image"]
models.append(
Model(
id=tf.id,
name=tf.id,
created=tf.created,
description=f"Tinfoil {tf.type} model",
context_length=tf.context_window,
architecture=Architecture(
modality=modality,
input_modalities=input_modalities,
output_modalities=output_modalities,
tokenizer="Unknown",
instruct_type=None,
),
pricing=Pricing(
prompt=input_price / 1_000_000,
completion=output_price / 1_000_000,
request=request_price,
image=0.0,
web_search=0.0,
internal_reasoning=0.0,
),
)
)
except Exception as e:
logger.warning(
"Failed to parse Tinfoil model",
extra={
"model_id": model_data.get("id", "unknown"),
"error": str(e),
"error_type": type(e).__name__,
},
)
return models
except Exception as e:
logger.error(
"Error fetching models from Tinfoil",
extra={"error": str(e), "error_type": type(e).__name__},
)
return []

302
tests/unit/test_tinfoil_integration.py Normal file
View File

@@ -0,0 +1,302 @@
"""Unit tests for Tinfoil direct integration.
Covers the EHBP usage-metrics header parser, the proxy header stripping, the
enclave URL override, and the TinfoilUpstreamProvider model fetching/forwarding
target logic.
"""
from __future__ import annotations
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from routstr.upstream.ehbp import (
_PROXY_ONLY_HEADERS,
_compute_ehbp_actual_cost,
_resolve_ehbp_target_url,
_strip_proxy_headers,
parse_tinfoil_usage_metrics,
)
from routstr.upstream.tinfoil import (
TinfoilModel,
TinfoilUpstreamProvider,
)
# ---------------------------------------------------------------------------
# parse_tinfoil_usage_metrics
# ---------------------------------------------------------------------------
class TestParseTinfoilUsageMetrics:
def test_full_header(self):
result = parse_tinfoil_usage_metrics("prompt=67,completion=42,total=109")
assert result == {
"prompt_tokens": 67,
"completion_tokens": 42,
"total_tokens": 109,
}
def test_without_total(self):
result = parse_tinfoil_usage_metrics("prompt=10,completion=5")
assert result == {"prompt_tokens": 10, "completion_tokens": 5}
def test_none(self):
assert parse_tinfoil_usage_metrics(None) is None
def test_empty(self):
assert parse_tinfoil_usage_metrics("") is None
def test_malformed(self):
assert parse_tinfoil_usage_metrics("garbage") is None
def test_missing_completion(self):
assert parse_tinfoil_usage_metrics("prompt=10") is None
def test_extra_whitespace(self):
result = parse_tinfoil_usage_metrics(
"prompt = 100 , completion = 200 , total = 300"
)
assert result == {
"prompt_tokens": 100,
"completion_tokens": 200,
"total_tokens": 300,
}
# ---------------------------------------------------------------------------
# _strip_proxy_headers
# ---------------------------------------------------------------------------
class TestStripProxyHeaders:
def test_strips_all_proxy_only(self):
headers = {
"x-routstr-model": "tinfoil-llama3-3-70b",
"X-Tinfoil-Enclave-Url": "https://inference.tinfoil.sh",
"X-Tinfoil-Request-Usage-Metrics": "true",
"Authorization": "Bearer secret",
"Ehbp-Encapsulated-Key": "abc123",
}
clean = _strip_proxy_headers(headers)
assert "x-routstr-model" not in clean
assert "X-Tinfoil-Enclave-Url" not in clean
assert "X-Tinfoil-Request-Usage-Metrics" not in clean
assert clean["Authorization"] == "Bearer secret"
assert clean["Ehbp-Encapsulated-Key"] == "abc123"
def test_all_proxy_only_headers_covered(self):
assert _PROXY_ONLY_HEADERS == {
"x-routstr-model",
"x-tinfoil-enclave-url",
"x-tinfoil-request-usage-metrics",
}
# ---------------------------------------------------------------------------
# _resolve_ehbp_target_url
# ---------------------------------------------------------------------------
class TestResolveEhbpTargetUrl:
def test_override_with_enclave_url(self):
result = _resolve_ehbp_target_url(
"https://default.example.com/v1/chat/completions",
"v1/chat/completions",
{"X-Tinfoil-Enclave-Url": "https://enclave.tinfoil.sh"},
)
assert result == "https://enclave.tinfoil.sh/v1/chat/completions"
def test_override_lowercase_header(self):
result = _resolve_ehbp_target_url(
"https://default.example.com/v1/chat/completions",
"v1/chat/completions",
{"x-tinfoil-enclave-url": "https://enclave.tinfoil.sh"},
)
assert result == "https://enclave.tinfoil.sh/v1/chat/completions"
def test_no_override(self):
default = "https://inference.tinfoil.sh/v1/chat/completions"
result = _resolve_ehbp_target_url(
default,
"v1/chat/completions",
{},
)
assert result == default
# ---------------------------------------------------------------------------
# _compute_ehbp_actual_cost
# ---------------------------------------------------------------------------
class TestComputeEhbpActualCost:
@pytest.mark.asyncio
async def test_no_usage_falls_back_to_max_cost(self):
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
result = await _compute_ehbp_actual_cost(None, model_obj, 100_000)
assert result == 100_000
@pytest.mark.asyncio
async def test_usage_parsed_and_clamped(self):
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
# The actual cost from calculate_cost will be small; we just verify
# it's clamped to min_request_msat at minimum.
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import CostData
mock_calc.return_value = CostData(
base_msats=0,
input_msats=10,
output_msats=20,
total_msats=30,
total_usd=0.0001,
input_tokens=67,
output_tokens=42,
)
result = await _compute_ehbp_actual_cost(
"prompt=67,completion=42,total=109",
model_obj,
100_000,
)
assert result == 30
assert result <= 100_000
@pytest.mark.asyncio
async def test_max_cost_data_falls_back(self):
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
with patch(
"routstr.upstream.ehbp.calculate_cost",
new_callable=AsyncMock,
) as mock_calc:
from routstr.payment.cost_calculation import MaxCostData
mock_calc.return_value = MaxCostData(
base_msats=0,
input_msats=0,
output_msats=0,
total_msats=0,
total_usd=0.0,
input_tokens=0,
output_tokens=0,
)
result = await _compute_ehbp_actual_cost(
"prompt=0,completion=0",
model_obj,
50_000,
)
assert result == 50_000
# ---------------------------------------------------------------------------
# TinfoilUpstreamProvider
# ---------------------------------------------------------------------------
class TestTinfoilUpstreamProvider:
def test_provider_type_and_defaults(self):
assert TinfoilUpstreamProvider.provider_type == "tinfoil"
assert (
TinfoilUpstreamProvider.default_base_url
== "https://inference.tinfoil.sh"
)
assert TinfoilUpstreamProvider.supports_ehbp is True
def test_transform_model_name(self):
provider = TinfoilUpstreamProvider(api_key="test")
assert provider.transform_model_name("tinfoil/llama3-3-70b") == "llama3-3-70b"
assert provider.transform_model_name("llama3-3-70b") == "llama3-3-70b"
def test_get_ehbp_forwarding_target_includes_usage_header(self):
provider = TinfoilUpstreamProvider(api_key="test")
model_obj = MagicMock()
model_obj.id = "llama3-3-70b"
model_obj.forwarded_model_id = "llama3-3-70b"
target = provider.get_ehbp_forwarding_target("v1/chat/completions", model_obj)
assert (
target.headers["X-Tinfoil-Request-Usage-Metrics"] == "true"
)
assert "v1/chat/completions" in target.url
def test_get_provider_metadata(self):
meta = TinfoilUpstreamProvider.get_provider_metadata()
assert meta["id"] == "tinfoil"
assert meta["name"] == "Tinfoil"
assert meta["fixed_base_url"] is True
def test_tinfoil_model_pricing_parses(self):
data = {
"id": "llama3-3-70b",
"context_window": 128000,
"created": 1721764788,
"pricing": {
"inputTokenPricePer1M": 1.75,
"outputTokenPricePer1M": 2.75,
"requestPrice": 0,
},
"endpoints": ["/v1/chat/completions"],
"type": "chat",
}
tf = TinfoilModel.parse_obj(data)
assert tf.id == "llama3-3-70b"
assert tf.pricing.inputTokenPricePer1M == 1.75
assert tf.pricing.outputTokenPricePer1M == 2.75
@pytest.mark.asyncio
async def test_fetch_models_parses_response(self):
provider = TinfoilUpstreamProvider(api_key="test")
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.raise_for_status = MagicMock()
mock_response.json.return_value = {
"data": [
{
"id": "llama3-3-70b",
"context_window": 128000,
"created": 1721764788,
"multimodal": False,
"pricing": {
"inputTokenPricePer1M": 1.75,
"outputTokenPricePer1M": 2.75,
"requestPrice": 0,
},
"endpoints": ["/v1/chat/completions"],
"type": "chat",
}
]
}
with patch("routstr.upstream.tinfoil.httpx.AsyncClient") as mock_client_cls:
mock_client = MagicMock()
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client.get = AsyncMock(return_value=mock_response)
mock_client_cls.return_value = mock_client
models = await provider.fetch_models()
assert len(models) == 1
assert models[0].id == "llama3-3-70b"
assert models[0].pricing.prompt == 1.75 / 1_000_000
assert models[0].pricing.completion == 2.75 / 1_000_000
assert models[0].context_length == 128000
@pytest.mark.asyncio
async def test_fetch_models_handles_error(self):
provider = TinfoilUpstreamProvider(api_key="test")
with patch("routstr.upstream.tinfoil.httpx.AsyncClient") as mock_client_cls:
mock_client = MagicMock()
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client.get = AsyncMock(side_effect=Exception("network error"))
mock_client_cls.return_value = mock_client
models = await provider.fetch_models()
assert models == []