# LLM Gateway: recorded benchmark evidence

This is a concise transcription of selected measurements in
`llm-gateway/docs/benchmarks/load-test-report.md`, dated 2026-09-30.
The benchmark was not rerun to create the portfolio case study.

## Environment and method

- Host: Apple M2 Pro, 12 cores, 16 GiB RAM; Docker Desktop on macOS.
- One Python worker per gateway replica; separate one- and two-replica scenarios.
- Fake provider: a 200 ms wait and known synthetic token usage; no real provider calls.
- Streaming scenario: 20 content chunks at 50 ms intervals, followed by usage and DONE.
- Campaign source revision: `2cb7ee35fb451a2fcad51513ca91325a03ac957d`.
- Open-loop k6 arrival-rate stages, with 30 seconds of excluded warmup.
- At least 3,000 measured requests per ramp stage unless generation was incomplete.
- One- and two-replica ramps ran three times; selected results are whole median ramps,
  not a mixture of the best percentiles from different repetitions.

Gateway overhead subtracts awaited provider operations from the gateway timing.
For streams, timing stops at the first nonempty body byte. Ordinary guardrail
scans remain included. These numbers are not full end-to-end model latency.
The verdict uses exact unrounded per-request observations, not histogram estimates.

## Selected ramp rows

| Replicas | Offered requests/s | Measured requests | Exact overhead p99 | Successful responses/s | HTTP errors | Dropped generator iterations |
|---|---:|---:|---:|---:|---:|---:|
| One | 50 | 3,001 | 7.161 ms | 49.844 | 0 | 0 |
| One | 100 | 6,001 | 7.199 ms | 99.672 | 0 | 0 |
| One | 200 | 12,001 | 24.435 ms | 199.332 | 0 | 0 |
| Two | 200 | 12,000 | 5.590 ms | 199.330 | 0 | 0 |
| Two | 400 | 24,001 | 31.620 ms | 398.661 | 0 | 0 |

SLO capacity was defined as the highest tested offered stage with exact overhead
p99 below 10 ms and HTTP errors below 0.1%. The selected one-replica ramp's
capacity was 100 offered requests/s; the selected two-replica ramp's was 200/s.
The one-replica 10/s and 25/s stages missed the overhead target, at 15.181 ms and
12.571 ms p99 respectively. Capacity is not a promise that every lower rate passed.

The highest fully generated error-eligible stages delivered 199.332 and 398.661
successful responses/s: a 2.000× scaling comparison. Their overhead exceeded 10 ms.
This throughput comparison is not the latency-qualified capacity comparison.

The two-replica 800/s stage delivered 532.693 successful responses/s, but the
generator dropped 14,985 iterations. Its 2.672× error-only scaling ratio is not
evidence that the system served every scheduled request at 800/s.

## Other recorded observations

- Streaming at 50 offered requests/s: exact overhead p99 9.364 ms.
- Approximately 5 KB personal-data scan/redaction at 50/s: exact overhead p99
  11.370 ms; target missed. No isolated guardrail-only duration was measured.
- Ten-minute soak at 50/s: 30,001 client successes, 30,001 fake-provider completions
  and 30,001 durable successful receipts after flushing. Exact overhead p99 7.242 ms.
- The soak had zero unexpected HTTP errors and zero dropped iterations, but its
  Redis error-counter delta was 2.0. HTTP success does not mean no dependency errors.
- RPM test: 600 requests/minute plus a 30-request burst allowance. Maximum observed
  admissions in any rolling 60 seconds: 629, 629, 629; bound: 630.
- Temporarily restoring the older rate counter made the regression test fail at
  631 admissions against the 630 bound. The GCRA implementation was restored.

## Limits of the evidence

- Laptop virtualisation and shared-host scheduling affect the result.
- Real providers introduce unmeasured network and model variability.
- No separately guardrail-excluded overhead distribution was established.
- Ten minutes of sampled memory does not establish absence of leaks.
- The bounded in-memory usage queue remains best effort, not a durable billing ledger.
- Token-per-minute limits remain approximate; the RPM bound requires available
  Redis with retained state and excludes fail-open outage behaviour.
- The full campaign measures its identified source/image. Later correctness or
  interface changes must not be described as fully benchmarked by this campaign.
- A load-test target and a completed campaign do not certify long-term availability.
