# Operations URL: https://antifailure.dev/docs/self-hosting/operations What to look at, what to do, and what not to do, when something is wrong at three in the morning. --- This page is written for the person who has just been woken up. It assumes you know nothing about the state of the system and have about ninety seconds of patience. Everything here has been run; nothing is aspirational. Setting the rotation up rather than firefighting inside it belongs on the [on-call page](/docs/self-hosting/on-call): who holds it, what an acknowledgement means, and what to do first for each class of page. The [status page](/docs/self-hosting/status-page) is what a customer reads while you read this one; it is not the pager and does not substitute for it. ## Create the first operator From this repository, with the Azure CLI signed into the deployment's subscription, run one command: ```sh just operator-init production ``` Use `staging` instead to target staging. The command reads that environment's checked-in deployment identifiers, verifies the exact resource group carries the Antifailure project tag, and opens setup in the running application. It does not create a container job or fetch the database password to your machine. The application image must include `af-operator`; an older image refuses the command rather than falling back to another account-creation path. Enter the operator email, display name and password. Password entry and its confirmation are hidden. The runtime uses its existing `AF_ADMIN_DATABASE_URL`, and the account can sign in at `/admin`. This is separate from GitHub customer sign-in. Setup refuses an existing root operator and never resets its password. For another container host, open a terminal in the serving container and run: ```sh af-operator init ``` Automation may supply the email and display name as arguments and pipe the password on standard input. A password or database URL is never a command argument. A successful Azure connection alone is not a successful setup: the wrapper also requires the runtime's completion acknowledgement. Always finish by signing in and opening an operator-only page. ## The first thirty seconds Three questions, in this order, because the answer to each changes which of the rest matter. **Is the control plane answering?** `curl -sf https://your-control-plane/health` returns `{"ok":true}`. If it does not, go to [The control plane is down](#the-control-plane-is-down), and note that environments are still working: nothing about `af up` needs the control plane, and engines are buffering their events to disk. **Is it the database?** `curl -s https://your-control-plane/metrics | grep af_http_requests_total`. A control plane that is up and failing everything almost always has a database it cannot reach. The process starts fine without one, because it does not connect until the first request. **Is it one organization or all of them?** `af_environment_outcomes_total` broken down by `code` answers this. One error code across many organizations is a platform fault. Many codes in one organization is that organization's repository, and is not your problem tonight. ## What the alerts mean Every rule in `observability/alerts/antifailure.rules.yml` links back here. They read the counters the control plane keeps itself, so they need a Prometheus scraping `/metrics`. The hosted control plane on Azure has a second, smaller set that needs no Prometheus and watches the platform rather than the process: the database, the replicas, the jobs, the certificate, and the service as a customer reaches it. Those have their own pages under [runbooks](/docs/self-hosting/runbooks), and each rule names its page in the notification it sends. Both sets are deliberately short. An alert nobody acts on trains everybody to ignore the ones that matter. ### ControlPlaneAvailabilityBudgetBurningFast Five percent of the month's error budget has gone in the last hour, and the last five minutes agree. The objective is 99.9 percent, which is forty-three minutes a month, so this is spending it fast enough to matter. Look at `af_http_requests_total` by `route`. One route failing is a bug in that handler and can usually wait for morning behind a rollback. Every route failing at once is the database, the pool, or a deploy. ### EnvironmentCreationFailing More than one environment in two hundred is failing to come up. Break `af_environment_outcomes_total` down by `code` first, before anything else. The code is an `AF-` reference and every one of them has a page under `https://antifailure.dev/docs/`; the page says what the failure is and what to do about it, which is faster than guessing from the count. ### TimeToPreviewOverObjective The slowest one in twenty environments is taking more than eight minutes to be reachable. Almost always one of two things: a golden that is being copied in full rather than branched, or a build cache that is not being hit. Neither is an outage and neither should be treated as one at three in the morning. ### IngestionIsLosingEvents The one to act on immediately. An engine treats a rejection as delivery and does not send that event again, so every rejected event is permanently lost and the environment it described may never advance again in the dashboard. The reason is on the ingestion response and in the API log for that batch. ### IngestionHasStopped No engine has reported anything for fifteen minutes. On a small installation this is usually nobody working, which is fine. The failure it is really watching for is invisible from here: engines that cannot reach ingestion buffer to disk and keep going, so a total ingestion outage looks exactly like a quiet night. If you have any reason to think somebody is working, treat this as an outage. ### RateLimitingIsRefusingRealTraffic `af_rate_limited_total` by `route`. One route is a limit set too low for honest traffic. Everything at once is one caller, and the per-organization kill switch is the tool for that. ## The control plane is down **Environments are not down.** This is the most important sentence on this page and the easiest to forget while being paged. `af up`, `af down`, `af test` and everything else work with no control plane at all. It was built that way deliberately: a preview environment that stops working because a web application is down would be a worse product than no dashboard. What is actually happening while it is down: - Engines buffer their events in memory and, when that fills or the command ends, to a spool directory under `.antifailure/spool` in the repository. The spool survives the process. The next command that runs against a control plane that has come back sends what the earlier ones could not, oldest first. - Nothing is lost until the spool exceeds its budget, at which point the oldest batches are dropped and the count is reported. You will see that reported rather than have to infer it. - `af env pull` fails, and says so with `AF-CPL-003`, which is the only user-visible consequence. So the recovery order is: bring the control plane back, and do nothing to the engines. They will catch up on their own. ## A deploy went bad and the automatic rollback did not fire `deploy/cd/deploy.sh` already rolls back on a failed post-promotion health gate, in the same run, before the gate exits. This section is for the failure that shows up after that: the gate passed, the run finished green, and the problem only became visible later, from a graph or a customer. Full procedure, including the case where a migration already applied and the revision you are about to restore may or may not still be compatible with it: [Upgrade and rollback, the manual path](/docs/self-hosting/azure#upgrade-and-rollback-the-manual-path). Do not skip that page's step on the migration; assuming compatibility instead of checking it is how a rollback becomes a second incident. ## Restoring the control plane database The commands below have been run. The recovery time this installation should expect is the one your own drill measured, not the one in any document. ### How much data an incident costs: the recovery point objective **Five minutes.** That is the recovery point objective for the control plane database, and it is Azure's number rather than one this project chose. Azure Database for PostgreSQL flexible server archives the write-ahead log continuously and documents the delay as up to five minutes, so a point in time recovery inside the primary region lands within five minutes of the failure. Five minutes of control plane writes is at most a handful of runs, verdicts and audit entries. Nothing in that window is a customer's data: raw snapshots, secrets and captured request bodies never leave the customer's cloud, and an engine that cannot reach the control plane buffers rather than dropping. **The recovery window is fourteen days**, which is `backup_retention_days` in `infra/terraform/modules/control-plane/variables.tf`. Azure allows 7 to 35 and its own default is 7. Fourteen is deliberate in both directions. Seven is not enough for the failure that actually needs a long window, which is not a lost server but a logical corruption nobody noticed: a bad migration on a Friday, found on the Monday after a week away, is already outside seven days and there is nothing to recover to. Thirty five is billed as backup storage every day for a window nobody has ever reached back into. Fourteen covers a fortnight, which is longer than anyone here has taken to notice a broken write. **A region loss costs up to an hour, and today it costs everything.** `geo_redundant_backup` defaults to `false`, so backups live only in the primary region and a region that is gone takes them with it. Turning it on gives a geo-restore with an RPO of up to an hour, because the copy to the paired region is asynchronous, and a geo-restore reaches the last backup that arrived rather than a second you choose: Azure does not offer point in time recovery from geo-redundant backups. Both of those are worse than the five minutes above, and both are enormously better than nothing. That default is correct for staging and wrong for production, and it is the expensive kind of wrong: backup redundancy can only be set when the server is created, so switching it on later means creating a new server and moving to it. Decide before the apply, not after. **None of this is the dump.** `af-control-plane-backup` is a second line with a different failure mode: it produces a file you hold, readable by any Postgres, which is what covers the case where the Azure subscription itself is the problem. Its recovery point is however long ago somebody last ran it, so it is worth a schedule of its own if you rely on it. ### Take a backup ``` af-control-plane-backup backup \ --url postgres://owner@host/antifailure \ --out /var/backups/antifailure ``` Three files come out: the dump, a roles file, and a manifest. All three matter. The **roles file** matters most and is the least obvious. `pg_dump` works on one database; roles live in the cluster. Restore a dump into a fresh cluster in another region and `antifailure_app` does not exist there, so every `GRANT` in the dump fails, `pg_restore` exits zero, and the application cannot connect to the database you just recovered. The roles file is what prevents that, and it is why a backup taken any other way is not enough. The **manifest** records what a restore has to reproduce: row counts per table, every policy, every table with row level security enabled and separately `FORCE`d, every privilege the application role holds, and the audit chain head. It records its own scope as well, which matters more than it sounds. All of those checks read the `public` schema, where every one of the control plane's tables lives. Anything outside it is listed in the manifest as unverified and reported by the restore and the drill as a table the check cannot speak for. That is not a restore failure and should not be read as one. It means the database grew somewhere this verification does not look, and somebody has to decide whether that table matters before the next real recovery. ### Restore it ``` af-control-plane-backup restore \ --url postgres://owner@newhost/postgres \ --database antifailure \ --dump /var/backups/antifailure/backup.dump \ --roles /var/backups/antifailure/backup.roles.sql \ --manifest /var/backups/antifailure/backup.manifest.json \ --app-password "$APP_PASSWORD" ``` It refuses a database that already exists. That refusal is deliberate: restoring over a live database is not a recovery, it is an outage with a different cause. Restore into a new name and switch the application over. It exits 3, and says which check failed, if the restored database does not match the manifest or does not isolate tenants. **Do not point the control plane at a database that exited 3.** `pg_restore` exits zero over a `GRANT` that failed because the role was missing, and over policies restored onto a table whose row level security it could not enable. Both produce a control plane that starts, answers every request, and isolates nothing at all. Nothing about it looks wrong from the outside. ### Rehearse it, on a schedule, before you need it ``` af-control-plane-backup drill \ --url postgres://owner@host/antifailure \ --out /var/backups/antifailure \ --database af_drill \ --app-password "$APP_PASSWORD" \ --report /var/backups/antifailure/drill.json ``` The drill backs up, restores into a throwaway database, checks it against the manifest, asks it through the unprivileged role to read another tenant's rows, drops it, and prints the recovery time it measured. Run it quarterly at least. A backup nobody has restored is a file. `--app-password` is not optional in practice. Without it nothing can connect as `antifailure_app`, so every check becomes a comparison of catalogue text against catalogue text, and all of that passes over a database that answers every query and isolates nothing. The drill treats a cross-tenant read it could not attempt as a failure and says so. The `restore` command says so and leaves the decision to you, because somebody recovering at three in the morning may not have the password to hand. It exits 3 when the restored database does not match or does not isolate, and 4 when the restore was sound and slower than a `--max-restore-seconds` budget you gave it. Two codes rather than one, because a backup that is not one and a runner having a slow morning are not the same finding and must not read as the same finding. This repository runs the drill against a scratch database every Monday at 04:00 UTC, in `.github/workflows/drill.yml`, which invokes the `drill` recipe in the `justfile` so that what runs unattended and what you can run by hand are the same command. Run `just drill` to run exactly that yourself: it starts a Postgres of its own, applies every migration, seeds two organizations so the cross-tenant read has another tenant to be refused, and holds the recovery time against a budget of 300 seconds. That budget is a backstop against a restore that has stopped working, not the objective below and not a performance target. What detects a regression is the series: the workflow publishes each measurement to the run summary and keeps it for ninety days. The number it prints is the **restore** time, not the whole run, because recovery starts from a backup that already exists. Counting the time to take one flatters the number by measuring work that has already happened when it matters. **Use your own number, not this one.** For scale only, all of it measured rather than estimated: on a continuous integration runner with nothing else on it, a control plane database holding a handful of organizations restored in under two seconds, and two consecutive runs of the same drill on the same runner reported 1.8 seconds and 0.6. On a development machine running a dozen other containers, the same restore took between 20 and 160 seconds. A factor of three between two runs an hour apart, and a factor of a hundred between two machines, is the point. A recovery time is a property of the hardware, the size of the database and what else is happening, so the only figure worth putting in an incident plan is the one your own drill measured on the machine you would actually recover onto. The suite prints its measurement on every run, so the number in front of you is never older than the last time anybody checked. The objective to hold it against is two hours, which is the recovery time this system is designed for. A drill that comes in well under it is not a reason to stop running the drill: what the drill really tests is whether the backup is one, and the timing is the part you get for free. ## Nobody can sign in Everything about access derives from GitHub. Who may sign in is a list of GitHub logins, membership comes from a GitHub App installation, and the role comes from GitHub. So a GitHub-side accident can lock every person out of a control plane that is otherwise running perfectly: the App deleted, its private key lost, the OAuth client secret rotated into the wrong variable, or an organization whose first sign-in happened while the App was broken and therefore has no owner. Fix GitHub first. The three that account for almost all of it: `AF_GITHUB_APP_ID` and its private key, `AF_GITHUB_CLIENT_SECRET` matching the OAuth App, and `AF_GITHUB_REDIRECT_URI` matching what the OAuth App has registered. The start-up log says which of these the process found. Getting sign-in working again is the real repair, and the command below is not a substitute for it. Reach for break-glass only when sign-in works and there is nobody inside the organization who can act, which means nobody holds `members.manage`. ``` af-control-plane-backup break-glass \ --url postgres://owner@host/antifailure \ --org acme \ --github-login somebody \ --role owner \ --reason "the App was deleted on 2026-08-30 and acme has no owner" \ --dry-run ``` `--dry-run` reads the current role and reports what would change, and writes nothing. Run it that way first; run it again without the flag to apply it. What it does and does not do, because both matter at three in the morning: - It sets one person's role in one organization, and nothing else. It issues no session and grants no login. It is not a way to be somebody. - **It cannot create an account.** It can only give a role to somebody who has signed in here at least once. If nobody ever has, what is broken is the OAuth configuration and no database write will fix it. - It refuses a change that would leave the organization with no owner, which is the state it exists to get out of. - It writes an audit entry, `member.break_glass`, with the reason you gave, the role before and after, and the login of whoever ran the command. That entry is inside the hash chain and cannot be quietly removed. Recording it is the whole reason to use this rather than `psql`, which would leave nothing behind. - The role is marked `manual`, so **Sync from GitHub** on the Members page does not undo the repair when GitHub comes back. Take it back by hand once it has. The `--url` must be a connection row-level security does not apply to: the cluster superuser, or a role with `BYPASSRLS`. Every tenant table is `FORCE ROW LEVEL SECURITY`, so the role that owns the schema is subject to the policies like anybody else. The command checks this before it does anything and says so, rather than updating nothing and reporting success. ## What not to do **Do not restore over the live database.** The tool refuses; do not work around the refusal. Restore beside it and switch. **Do not use break-glass to add yourself to a customer's organization.** It records who ran it and why, in a log the customer can export. It is for restoring access somebody already had, not for acquiring access nobody granted. **Do not run `af env prune --older-than 0` to clean up during an incident.** It removes every environment on the machine, including ones somebody is using to debug the incident. `af down --branch ` takes one. **Do not delete a golden to reclaim space while environments are running.** `af golden gc` already refuses to collect a version an environment came from, and reports `AF-DB-005` when asked to. That refusal is the feature. **Do not disable a row level security policy to unblock a query.** It is the only thing separating one customer's data from another's, and the drill above will tell you it is missing long after somebody has read something they should not have. ## Collecting evidence before you change anything ``` af support bundle ``` Logs, decisions, manifest, and doctor output, redacted, with a listing of exactly what it included. Take one before you start changing things, because the state that explains the incident is usually the first thing a fix destroys. For one environment specifically: ``` af status af logs web af doctor ``` `af doctor` runs a check for each thing that can stop a run, and every one of them carries a remediation. It is the fastest way to find out that the thing you are debugging is a Docker daemon that is not running, or that this machine is still holding environments from runs that failed days ago. ## Load testing the control plane itself `af load` shapes traffic against an environment `af up` built; nothing before this pointed it at the control plane's own API, which is the one service in this product that has never had its own load generator run against it. `engine/cmd/loadcp` does, using the same `engine/internal/load` package `af load` does, against a URL instead of an af-managed environment: ```sh go run ./cmd/loadcp -url https://app.dev.antifailure.dev -duration 1m -scale 1 ``` The bundled profile is not measured production traffic; none has been captured yet, and there is nowhere in this product's own load package to point at the control plane's access log until there is one. Each route's weight is instead its own declared ceiling from `web/apps/api/src/limits.ts`, the number the rate limiter already enforces per caller. The profile says so: its `source` field reads `declared_limits`, not `production`, the same honesty `internal/load` itself applies to a shape nobody supplied. **What a real run found.** Built and run once against a real local instance, schema migrated, serving from an actual Postgres, not a fake: at half the combined declared rate (92 requests a second, one caller, `-scale 0.5`), p95 latency climbed from 0.5 seconds to 3.2 seconds over a 31 second run, achieving 37 requests a second against a target of 92, with `/readyz` carrying the worst tail at up to 4.9 seconds. No request was rejected by the rate limiter at any point in this run; the connection pool queued first. That run was on a laptop reporting a load average over 75 from other work sharing the same machine at the time, which is exactly the caveat this project's own [disaster recovery timings](#rehearse-it-on-a-schedule-before-you-need-it) already carry: a latency number is a property of the hardware and what else is running on it, not a portable fact about the code. What is portable is the finding underneath it, which is worth checking again on quiet, dedicated hardware before it informs a real capacity decision: on this run, the database connection pool (`AF_POOL_MAX`, ten by default) became the limiting factor before the per-caller rate limits did, for a single caller sending across every route at once. An operator sizing a real deployment should raise `AF_POOL_MAX` to match expected concurrent callers rather than assuming the rate limiter is the only ceiling in the system. ## Where the numbers come from `GET /metrics` on the control plane, in the Prometheus text format. It reads no tables: tenancy here is row level security, so an aggregate across every organization would need a role that can read every organization's rows, and creating one in order to draw a graph would put the strongest read in the system on the least important path and leave it there being scraped every fifteen seconds forever. Everything exposed is a counter the process kept itself, and several replicas each expose their own for Prometheus to sum. The dashboard is `observability/dashboards/control-plane.json`, importable as it is. Its panels and the alert rules are both checked against the exporter by a test, because an alert on a metric that does not exist fires never and a panel on one draws an empty graph, and an empty graph reads as a quiet system rather than as a broken dashboard.