Centralising model access
Integrating several model providers creates work that each application would otherwise repeat: credential management, model permissions, usage accounting and failure handling. These decisions also need to hold across teams and gateway replicas.
I built the gateway to put those decisions in one place. An application sends an OpenAI-shaped request with a gateway-issued key. The gateway checks its team’s permissions, calls an allowed provider, and records the usage. Applications don’t receive the provider credentials.
The design had to handle disconnected streams, missing token counts and concurrent admission across replicas. Routing decisions also had to respect a team’s model and data-residency policies during retries and fallback.
Two paths through the system
The application path and the administrator path have different responsibilities. Applications use the public model API. The console talks to a separate private admin API through its Next.js server. The browser never needs the admin credential in readable form.
OpenAI-compatible client
Gateway-issued team key
Authenticate · Check policy
Inspect input · Admit request
Choose provider · Record usage
Groq · DeepSeek · Gemini
OpenAI · Z.ai · NVIDIA
- Redis ↔ Gateway
- Shared rate and budget counters, concurrency leases and encrypted response cache.
- PostgreSQL ↔ Gateway
- Organisations, teams, hashed keys, policies, usage receipts and audit events.
Postgres holds the records that need to survive a restart. Redis coordinates fast decisions across replicas. I avoided putting a separate copy of a team’s budget in every process: adding a replica should not accidentally give that team a second spending allowance.
The code follows the same boundaries. HTTP handlers parse a request and call the layer below. Provider adapters deal with provider differences. Typed schemas describe the shared request and response format. Accounting, limits and guardrails each have their own modules rather than accumulating inside one large route handler.
A shared request path across providers
Streaming and connection lifecycle
I used the OpenAI API shape so existing clients could point at the gateway by changing their base URL. The request path adds request IDs, metadata-only logs, timeouts and consistent error responses around the provider call.
Streaming forwards chunks as they arrive and closes the upstream connection when the client disconnects. This keeps cancellation tied to the provider connection and avoids buffering a complete answer before returning it.
Error mapping needed care too. A provider’s 401 usually means the gateway’s provider key is wrong, not the application’s gateway key. Returning that same 401 would tell the caller to fix the wrong credential. The gateway reports it as an upstream failure instead.
Canonical schemas and stream parsing
I replaced raw pass-through with Pydantic models for chat, tools, structured output and embeddings. Requests are strict, so a misspelled field gets a useful error before any paid call. Provider responses are tolerant: an added field should not break the gateway if the fields it uses are still valid.
A streaming event can arrive across several byte reads, even in the middle of a Unicode character. The Server-Sent Events (SSE) decoder rebuilds complete events before validating them. An answer ends with a clear completion marker or an error; a provider that stops halfway through is not reported as a finished answer.
Provider-specific options live in a named provider_options block. Only the serving provider’s options are forwarded. That block cannot replace canonical fields such as messages, which would let callers hide text from later validation and guardrail checks.
Provider adapters
Groq, DeepSeek, Gemini and OpenAI use separate adapters with shared transport and stream handling. A model name such as groq/openai/gpt-oss-120b selects Groq at the first slash; the rest is the provider’s model ID.
Each provider has its own connection pool. A slow provider can fill its own pool without taking all the others’ connections. I also kept capability checks evidence-based: an explicitly unsupported endpoint is rejected locally, but a parameter is not rejected just because its documentation is incomplete.
POST /v1/chat/completions
Authorization: Bearer YOUR_TEAM_API_KEY
{
"model": "groq/openai/gpt-oss-120b",
"messages": [{"role": "user", "content": "Explain a circuit breaker."}],
"stream": true
}
Tenant isolation and usage accounting
Organisations, teams and virtual keys
A tenant is an organisation using the service. Teams sit inside it, and application keys belong to a team. Every model request must authenticate before the gateway reads its body. Health endpoints remain separate from that requirement.
A new key is shown once. Postgres stores a public identifier and a hash of the secret, mixed with a server-only pepper, rather than a recoverable copy of the key. Provider secrets come from environment variables or restricted mounted files.
A verified-key cache removes a database lookup from repeated calls. Its default 30-second lifetime also bounds the delay before revocation or a cached policy change reaches each replica.
Prices and usage receipts
The reviewed model catalogue holds prices with effective dates. A request uses the rate valid when it started, and its receipt keeps the catalogue version. Updating tomorrow’s price should not rewrite yesterday’s usage.
Costs use decimal arithmetic in Python and Postgres, not floating-point money. Missing token usage stays unknown. A disconnected stream may still have incurred a charge, so a missing final usage chunk is not a reason to record a free request.
I moved receipt writes into a bounded in-memory queue with a background batch writer, keeping Postgres inserts off the response path. Queue saturation or process loss can drop records, so this is best-effort usage accounting. Drops are counted and logged. A billing ledger would need durable ingestion and reconciliation.
Shared limits and budgets
Redis enforces requests per minute, recorded tokens per minute, concurrency and monthly budget checks. Atomic Lua scripts prevent two replicas from both reading the same remaining allowance and spending it twice. Concurrency uses renewable leases so a crashed process does not leave a slot occupied forever.
Budget amounts are integer pico-dollars, a trillionth of a dollar. Durable receipts can rebuild a missing counter. Operators choose whether a Redis failure should allow traffic to continue or reject it until cost controls are available.
These controls have limits. Actual token usage arrives after the response, so in-flight work can overshoot token limits and budgets. Requests-per-minute enforcement was also initially approximate. The load tests later exposed that weakness and led to a different algorithm.
Failure handling and routing policy
Retries, circuit breakers and approved fallback
A timeout does not prove a model stopped working. Repeating the call may buy a second answer. I limited retries to selected connection failures and overload or server responses; read-timeout retries remain off unless an operator accepts that risk.
Retry waits include randomness, a shared per-request deadline and a per-process retry budget. A circuit breaker stops calling a repeatedly failing provider for a short interval, then admits one probe. Breakers are local to each replica, so they still operate when Redis is unavailable.
Fallback is reviewed rather than automatic across every provider. It can send a prompt to another company, with a different model and processing location. The catalogue lists allowed alternatives, and the client can disable fallback. Once a provider stream has opened, a failure ends it; the gateway does not splice a new model’s answer onto the old one.
Observability and an audit trail
Prometheus metrics show traffic, latency, failures and receipt losses. OpenTelemetry traces break a request into authentication, admission and provider attempts. Logs connect those records through request IDs. None of these stores prompts, answers, embeddings or keys.
Administrative changes have a separate audit chain. Each event includes the previous event’s hash, and the setting change and its audit event commit in the same transaction. That makes later edits detectable, but not impossible: a database owner could rewrite the chain. Stronger protection would need a trusted checkpoint kept elsewhere.
Model access, aliases and weighted routing
An organisation sets the outer model-access boundary; a team can narrow it. The gateway resolves a stable alias such as fast to a concrete model before checking permission. Every fallback destination is checked again. An outage must not turn into a permission bypass.
Weighted routing allows a reviewed split between candidate models without changing application code. The actual serving model and requested alias remain on the receipt. Each request makes its own weighted choice; ten requests are not guaranteed to produce a perfect nine-to-one split.
Caching, guardrails and data residency
A team-isolated response cache
Repeated embeddings are a useful caching case. Chat caching requires an explicit opt-in because a different answer can be intentional. Cache lookup happens after authentication, model permission and request-rate admission. A hit avoids a provider call but is still a gateway request.
Cache keys include the team, concrete model and catalogue version. The request content is hashed; stored answers are encrypted and bound to their cache address. Moving one team’s ciphertext under another team’s key should not make it readable there.
Single-flight coordination lets identical requests on one replica share a completed result. Cache reuse requires an exact match within the same team; streams, semantic similarity and cross-team reuse are excluded.
Guardrails and data residency
Deterministic detectors block recognised secrets and can redact personal data. A repeated email becomes the same placeholder within a request, and a request-local mapping restores it in the answer. The mapping is never stored in logs, receipts or Redis.
Organisation and team rules use the strictest action. Residency policies also filter the real model destinations, including aliases and fallbacks. A provider’s storage promise is not automatically an inference-location promise; unknown and global regions stay labelled as such. An EU-only policy currently has no eligible reviewed destination.
Pattern matching cannot recognise every name, encoded secret or obfuscated address. Images, audio and files are not inspected by these text detectors. Streaming adds another boundary: newly generated sensitive output can be detected at the end, but bytes already delivered cannot be taken back.
The tests changed the design
A private management API
Before building the console, I added a separate admin listener. Application keys and admin keys are not interchangeable. Platform administrators can manage all organisations; organisation administrators are checked against their own scope on every resource lookup.
The CLI and HTTP API use the same administration service. A retried creation uses an idempotency key rather than creating a second resource. Newly issued application secrets still appear only in the first response. A usage dashboard reads Postgres through a restricted database role with no access to the key tables.
Isolating gateway overhead
The performance question was how much delay the gateway adds. I used a fake provider with a fixed 200-millisecond wait and known token counts, then tested one and two replicas with k6 on an M2 Pro laptop through Docker Desktop. This isolated the gateway from model variability and avoided paid calls.
The target was gateway overhead below 10 milliseconds at p99, with fewer than 0.1% HTTP errors. Overhead excludes awaited provider operations; for streams, timing stops at the first nonempty body byte.
The first campaign was not enough. A low-rate stage had only about 60 samples, and coarse histogram buckets could not support a precise verdict near the 10-millisecond target. The amended campaign excluded warmup, collected at least 3,000 requests per ramp stage and used exact per-request overhead values.
| Replicas | Offered requests/s | Overhead p99 | Under 10 ms? |
|---|---|---|---|
| One | 100 | 7.199 ms | Yes |
| One | 200 | 24.435 ms | No |
| Two | 200 | 5.590 ms | Yes |
| Two | 400 | 31.620 ms | No |
At the fully generated, error-eligible stages, successful throughput roughly doubled: 199 to 399 responses per second. Those higher-throughput stages did not meet the overhead target. The selected one-replica run met it at 50–100 offered requests per second, but missed at 10–25 and at 200. A 5 KB personal-data redaction scenario also missed, at 11.370 milliseconds p99.
One higher stage produced a tempting 2.672× scaling ratio, but the generator dropped 14,985 scheduled iterations. Those requests never reached the gateway. I used the fully generated stages for the headline comparison rather than presenting missing traffic as capacity.
The rate-limit bug the load exposed
The original weighted-minute counter could admit too many requests around a boundary. It was replaced with GCRA, an algorithm that assigns each admitted request a time slot. Redis supplies the shared clock and makes the check atomic.
At 600 requests per minute with a burst allowance of 30, the expected rolling-minute bound was 630. The amended runs observed a maximum of 629 in all three repetitions. A regression test included a late-minute burst; temporarily restoring the old counter made it fail at 631. That was stronger evidence than a test that only sent evenly spaced traffic.
The ten-minute soak reconciled 30,001 successful client responses with 30,001 fake-provider completions and 30,001 durable successful receipts after flushing. No receipts were lost in that run.
Measurement scope: these results cover the 30 September source revision on an M2 Pro through Docker Desktop, using a controlled provider. Guardrail work remains included. The campaign establishes capacity at its tested stages; production capacity and long-term memory behaviour need separate measurements.
The admin console and its security boundary
Server-side credentials
The Next.js server acts as the console’s backend for frontend. The browser calls same-origin routes; server-only code calls the private admin API. The admin credential is sealed in an encrypted, Secure, httpOnly session cookie and stays out of React props and readable browser responses.
Every write checks the configured browser origin. A nonce-based Content Security Policy limits scripts, and authenticated responses are not cached. The gateway remains the authority on roles and organisation scope; hiding a control is not an access check.
Money arrives as decimal strings and stays exact through BigInt formatting. Unknown costs appear as unpriced rather than zero. Charts have labelled axes and table alternatives. Browser tests scan responses for credential exposure, including the key-creation flow where a new application secret may be returned once.
Policy editing and concurrent updates
An inherited policy is easy to misread. The editors show the organisation rule, the team override and the effective result together. Removing an override means inherit; saving an empty allowlist means allow nothing. Those are separate actions.
A version returned by the API travels back in an If-Match header on save. If someone else edited the policy, the API rejects the stale save and the console keeps the draft. Changes that weaken protection or deny all access require a consequence confirmation. A cache purge also asks for the exact organisation or team name.
Provider integration and deployment boundaries
Z.ai and NVIDIA-hosted models
Adding providers exercised the boundaries already in place. NVIDIA can host a model made by another company, so the routing prefix identifies the host, not necessarily the model maker. Host-specific prices and residency evidence cannot simply be copied from the maker.
NVIDIA’s trial models remain explicitly unpriced. Known tokens can be recorded while dollar cost stays unknown; a dollar budget cannot limit a cost the gateway does not know. Budget-limited teams need a model policy that excludes those destinations.
Queued Kimi requests can return HTTP 202 before a result is ready. The adapter submits once and polls the documented result endpoint within the original deadline. A polling failure must not turn into a second billed submission. Queue waits use provider-specific timeouts.
Request inspection and analytics
The Requests screen groups provider attempts by request ID, exposing retries and fallback through metadata. Filtered CSV export, model information and provider health support investigation without retaining conversations.
Analytics reports recorded attempt duration, including provider waiting. That is different from the gateway-overhead measurement in the load test. Missing first-byte times and costs remain unknown. The calendar uses UTC and preserves the database’s microsecond boundaries so a date filter does not quietly lose the last fraction of a day.
Deployment boundaries
The hosting platform’s service topology would have exposed the admin API if I deployed it as a separate web service. I packaged the hosted console, gateway and test provider in one non-root container, binding internal endpoints to loopback and exposing only the console port.
That hosted configuration uses one instance. The gateway’s shared-state design supports multiple replicas, as exercised separately in the load tests. Viewer access is enforced by the API and its organisation checks; the browser interface does not grant privileges.
Remaining engineering work
Billing-critical use requires a durable receipt path with idempotent consumers and reconciliation against provider invoices. Hard budget enforcement also needs reservations for work still in flight, alongside completed receipts.
The next performance investigation is to correlate slow requests with authentication-cache misses, Redis waits and writer activity, then repeat the campaign on isolated hardware. The low-rate outliers and redaction scenario need an explanation before changing the runtime or adding faster serialization.
Architecture and benchmark records
The architecture notes and decision records document the implementation and its tradeoffs. Measurements come from the amended 30 September 2026 load-test campaign; the source revision and test conditions are recorded with the results. Console screenshots were captured on 2 October and use synthetic records.
Which records informed each part?
- Protocol and provider boundaries: architecture steps 1–3 and ADRs 0001–0005.
- Tenancy, accounting and admission: steps 4–6, the step 12b amendment and ADRs 0006–0011.
- Recovery, policies and privacy: steps 7–11 and ADRs 0012–0021.
- Admin API, measurements and console: steps 12–13 and ADRs 0022–0025.
- Provider integrations and deployment: steps 14–16, ADRs 0026–0030 and the step 16c report.
Download the benchmark notes for the selected rows, measurement definitions and limitations.
The full benchmark report and architecture notes are available with the public source.