Traffic shaped like production's, sent at the twin.
The engine reads your own production traffic, keeps the mix of routes it actually served, and sends that mix at the twin. Every route is unsafe until the manifest names it, and a trace export arrives carrying production's own p95 per route, so the answer is a comparison rather than a score.
Production-shaped route comparison
- source
- OTel
- OTLP / JSON
- achieved
- 17.8/s
- reported, not target
- baselines
- 3 of 4
- shown routes
- threshold
- +25%
- p95 increase
Route mix + p95 comparison
weighted arrivals · baseline marker in black · 0 to 450ms scale
- GET /api/subscriptions
largest p95 increase
breach+129%share18%p95412ms/ 180ms - GET /settings/billing
production baseline compared
within+12%share34%p95168ms/ 150ms - GET /
production baseline compared
within+7%share27%p9544ms/ 41ms - POST /api/search
allowed route without baseline
unscoredno baselineshare9%p95228ms/ no base
GET /** · POST /api/search
POST /billing/upgrade · POST /api/payments/intent
IllustrativeA shaped run, to show the format. The routes, latencies and shares are written. The columns, the sort order and the thresholds are the ones af load produces.
The mix is the point. A load test that hammers one endpoint proves the endpoint is fast, which nobody doubted.
What breaks under real traffic is the mix: the page nobody thinks about that is nine percent of requests, and the endpoint that is fine alone and holds a lock the hot path wants. So the traffic is a weighted mix read from what production actually served.
The mix, not one endpoint
Routes are weighted by the share of requests production actually served them. A flat mix proves the endpoint you already trusted is fast.
Poisson arrivals
Requests arrive in clumps, the way real ones do. Evenly spaced arrivals hide the queueing that the change is about to make worse.
Deterministic per seed
The same seed sends the same requests in the same order, so two runs are comparable and a difference belongs to the change.
The achieved rate, reported
The report carries the rate the generator managed, not the rate it was asked for. Reporting the target is how a load test says everything was fine while the queue grew.
Unsafe until named
No route is sent until the manifest names it safe. With no allowlist the default is read-only GETs under the root.
Compared, not scored
Each route is measured against the p95 production serves it in. The answer is a delta, never an absolute capacity claim.
Two sources, and only one of them carries a baseline. A trace export carries a latency. A log line does not.
otel reads an OpenTelemetry trace export in OTLP/JSON, the file a collector's file exporter writes. access_log reads a combined format log. Both are read out of the repository, so no credential and no outbound call decides what traffic gets sent. A span has a start and an end, so a shape read from traces arrives with production's p95 for each route in it, which is what p95_increase compares against. A combined format line carries no duration, so an access log gives the mix, the weights and the arrival rate, and no baseline to measure a regression against. Setting p95_increase under the log is refused when the manifest is read, rather than accepted and quietly skipped.
There were four sources once. Two of them existed only in the schema and were refused when a run reached them, which is worse than not offering them: a key you can set that cannot work reads as a broken product rather than an unfinished one. They are gone, and anything unrecognised is refused when the manifest is read, before anything is built.
load.source: There is no load source called "datadog". The sources that read traffic are otel, an OpenTelemetry trace export, and access_log, a combined format log. Both take source_config.path.
load.thresholds.p95_increase: The load source is access_log and p95_increase is set. A combined format log line carries no duration, so every route read from one arrives with no baseline and this threshold can never fire.
Manifest compilation
workload manifest
repositoryload: enabled: true source: otel source_config: path: traffic/production.otlp.json scale: 0.05 duration: 2m safe_routes: ["GET /**", "POST /api/search"] unsafe_routes: ["POST /api/payments/**", "DELETE /**"] thresholds: p95_increase: 0.25 error_rate: 0.01source capabilities
- route weightsyesyes
both sources can shape the route mix
- arrival rateyesyes
achieved rate is measured after compilation
- p95 baselineyesnone
baseline comparison requires traced latency
- p95_increasevalidrefused
access log manifests cannot enforce p95 increase
AF-MAN-002 · access_log + p95_increase is refused before anything is built.
compile path
no outbound lookup- 01read
repository file
- 02validate
source + threshold
- 03shape
mix + arrivals
- 04send
allowlisted routes
A route with no baseline is never a breach. Comparing against nothing and calling the answer a regression is how a check becomes noise.
Thresholds are deltas against what production serves, never absolute numbers: an absolute limit fails on a slow runner and says nothing about the change. A route the export saw fewer than twenty times is listed with its latency and no verdict, because a percentile made of three numbers is noise. When no route in a run has a baseline, the threshold measured nothing and the run says so instead of reporting a clean p95.
Know what happens before you deploy.
Create a disposable production twin for every risky change. Catch migration failures before they reach customers.