On-call

The rotation

One person, holding the pager continuously, until this page names a second one.

With a second person, the rotation is a fixed weekly handoff: whoever is on call through Sunday hands off Monday morning, in a message naming which alerts fired that week and what is still open, not just “nothing happened”.

What an acknowledgement means

Acknowledging a page means: I have seen this, I am looking at it now, stop paging anyone else about it.

Concretely, an acknowledgement means, within the next few minutes:

• Read the first thirty seconds of the operations page and answer its three questions.
• Say, somewhere a second person could read it, what you found. A one-line status is enough: “/readyz is failing, looks like the database, digging in.”
• Decide whether this is something you can carry alone or something that needs the second escalation below, before you are an hour into it and out of runway.

An unacknowledged page after the escalation window is treated as a missed page.

When to wake somebody

Three questions, and any one of them being true is enough on its own.

Is a customer’s data at risk? A row-level security failure, a masking failure that let raw data leave the boundary it is supposed to stay inside, a credential that may have leaked. Wake somebody now, and do not wait for a second opinion on whether it is bad enough. docs/plan/prod_guide.md has the incident that reshaped how this project applies infrastructure changes.

Is the whole control plane down, not one organization? The operations page’s second and third questions tell you which: many error codes in one organization is that organization’s own repository and can wait for morning. One error code across many organizations, or /readyz failing outright, is a platform fault and does not wait.

Has IngestionIsLosingEvents fired? Named explicitly because it is the one alert the operations page marks “act on immediately”: an engine that has an event rejected treats it as delivered and never sends it again, so every rejected event is gone for good and the environment it described may never advance in the dashboard again. There is no fail-open behind this one.

Everything else on the alerts page carries its own judgment call in its own section; read the alert’s own runbook before deciding it can wait, rather than guessing from the name.

What to do first, by class of page

The control plane will not answer at all. Read The control plane is down before doing anything else. af up, af down, and every environment already running keep working with no control plane at all. Bring the control plane back; do nothing to the engines, they catch up on their own.

A deploy just went out and something looks wrong. Check whether the automatic rollback already fired: a failed post-promotion health gate moves traffic back within the same CD run and the run’s summary says so. If it did not, and the run finished green, the failure showed up after the gate stopped watching. Follow Upgrade and rollback, the manual path, in order, including the migration compatibility check in its fourth step. Do not skip to “just roll the code back” before reading that step: a migration that is not backward compatible makes a code rollback the wrong fix, not the safe default.

A specific alert fired. Its entry under What the alerts mean is the runbook. Read it before touching anything.

Something feels wrong and no alert has fired. Trust it, and start from the first thirty seconds anyway. An alert is a threshold somebody guessed in advance; a person noticing something first is not a false alarm just because nothing crossed the line yet.

Collecting evidence before you fix anything

af support bundle for one environment, and for the control plane itself, the steps under Collecting evidence before you change anything.