The control plane is unreachable

Alert: unreachable. Severity 0. The service is down for customers.

An availability test asked https://app.antifailure.dev/readyz from three Microsoft managed locations and at least two of them failed inside fifteen minutes. Each agent retries a failed request before reporting it, so this is already not a single dropped packet, and two separate locations agree.

What it has already ruled out

The probe asks for the customer’s name over TLS, so it exercises DNS, the custom domain binding, the certificate, the ingress and the application. Any one of those is enough to fire it. That breadth is the point and it is also why the first job is to narrow it.

/readyz is not /health. It takes a connection out of the pool the application serves with and asks the database a question, and it answers 503 when the database does not. A 503 here is the application telling the truth.

Thirty seconds, in this order

curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' \
  https://app.antifailure.dev/readyz
curl -sS https://app.antifailure.dev/readyz
dig +short app.antifailure.dev

No DNS answer. The CNAME is gone or the zone is broken. It lives in the af-web resource group, not in the control plane’s, so a change there is the first thing to look at.

A TLS error. Go to the certificate.

503 with a reason. The database. Go to the database is not answering.

404 or an Azure error page. The custom domain binding, or traffic is on a revision that is not serving. Check what is actually serving:

az containerapp ingress traffic show -n afcpprod-app -g af-cp-prod-centralus -o table
az containerapp revision list -n afcpprod-app -g af-cp-prod-centralus \
  --query "[?properties.active].{rev:name,healthy:properties.healthState}" -o table

Nothing answers at all. Ask the generated address, which skips DNS, the binding and the certificate in one step:

az containerapp show -n afcpprod-app -g af-cp-prod-centralus \
  --query properties.configuration.ingress.fqdn -o tsv

If that address is healthy and the custom name is not, the fault is in the four resources in infra/terraform/modules/control-plane/domain.tf and nowhere else.

What not to do

Do not roll back before reading what is serving. In Multiple revision mode the previous revision is still running at zero percent. Moving traffic back to it is one command and a few seconds. Redeploying is minutes, during which the broken revision is still taking requests.

Do not assume a deploy caused it without checking. This alert fires for a certificate, a DNS record and a database, none of which a deploy touches.

Environments are not down. Customers running af up in their own continuous integration are unaffected, and their engines buffer events to disk until this comes back. The operations page explains what that recovery looks like, and it needs nothing from you.