Standing up production

The production control plane is one terraform apply and fifteen steps in a browser or a shell. The order matters: several fail if done early.

Read Azure first. Everything on that page about policy, regions, the Key Vault name and the revision mode trap applies here and is not repeated.

What Terraform owns

infra/terraform/stacks/control-plane/production.tfvars is the whole configuration and every value in it says why it differs from staging. One apply produces the resource group, a zone redundant Postgres with geo redundant backups, the Key Vault, the bootstrap and maintenance jobs, the application on two replicas, the DNS records for app.antifailure.dev, the managed certificate, the custom domain binding, and ten alert rules with an action group.

What Terraform cannot own, and why

The GitHub App’s private key and webhook secret. GitHub mints the key once and shows it once, so Terraform can neither create it nor recreate it. The module reads both from Key Vault with a data source instead, which is also why setting github_app_id before those secrets exist fails at plan rather than at the first delivery.

The OAuth App’s client secret. Same reason. Terraform seeds a placeholder once and then carries ignore_changes on the value, so rotating it with az keyvault secret set stays true.

The managed certificate’s binding to the custom domain. Not a policy decision, a circular one. Azure refuses to issue a managed certificate for a hostname that is not already bound to an app in the environment, and refuses RequireCustomHostnameInEnvironment if you ask the other way round, so the binding cannot name a certificate that cannot exist until the binding does. Terraform adds the hostname with no certificate, Terraform creates the certificate, and one az containerapp hostname bind closes the loop. The ignore_changes on the custom domain is what stops the next apply undoing it. Step 6 below is that command.

Role assignments outside this stack’s group. The DNS zone is in af-web. A stack that could grant itself write access to another group’s resources would defeat the point of scoping it.

The federated credential and the deployment approval rule. Both are how the repository proves who it is, and both are deliberately outside anything a pull request can change.

The checklist, in this order

1. Give production its own Terraform state

This is the step that can destroy staging, and it is first for that reason. The stack directory is shared: staging.tfvars and production.tfvars sit side by side and the backend is configured at init time. Running terraform apply -var-file=production.tfvars in a directory that was initialised against staging’s state produces a plan that destroys staging and creates production, and it will look like a very large diff rather than like a mistake.

So production gets its own backend configuration with a different key:

cd infra/terraform/stacks/control-plane
cat > backend.production.hcl <<'EOF'
resource_group_name  = "af-tfstate-eastus"
storage_account_name = "<the state account>"
container_name       = "tfstate"
key                  = "control-plane-production.tfstate"
use_azuread_auth     = true
EOF
terraform init -backend-config=backend.production.hcl -reconfigure

backend.hcl and backend.production.hcl are both ignored by git, because the storage account name is an identifier this repository does not carry.

Read the first line of every plan, and then read its exit status. A plan against the right state adds roughly forty resources and destroys nothing, and anything with destroys in it is the wrong state.

The exit status is the separate check, and it is the one that has caught things here. This stack has twice produced a plan that printed in full, ended with its own 0 to destroy summary, and then exited non-zero. Once for an output that carried a provider-sensitive value without declaring itself sensitive, which Terraform refuses while evaluating outputs and therefore after the whole diff has been printed. Once for the managed certificate’s RequireCustomHostnameInEnvironment. Both look exactly like a plan that worked, and the only thing that tells them apart from one is echo $?.

2. Check the region, before anything else

go run ./tools/azguard region centralus

It fails closed. A region it cannot get an answer about is refused.

3. Grant the deploying identity access to the DNS zone

The records for app.antifailure.dev are created in the antifailure.dev zone, which lives in af-web. Whoever runs the apply needs to be able to write there.

az role assignment create \
  --role "DNS Zone Contributor" \
  --assignee-object-id "$(az ad signed-in-user show --query id -o tsv)" \
  --assignee-principal-type User \
  --scope "$(az network dns zone show -g af-web -n antifailure.dev --query id -o tsv)"

Subscription Owner already covers this. Run it anyway if the apply is done by a service principal rather than by a person.

4. Decide who gets paged

The addresses are not in this repository and are passed as environment variables. Enabling alerting with no receiver fails at plan, on purpose.

export TF_VAR_alert_emails='["you@example.com"]'
export TF_VAR_alert_sms_country_code='1'
export TF_VAR_alert_sms_number='5551234567'

5. Plan, price it, apply

terraform plan -var-file=production.tfvars -out=plan.tfplan
terraform show -json plan.tfplan > plan.json
go run ./tools/cost estimate --plan plan.json --pricing infra/pricing.yaml --budget 450
terraform apply plan.tfplan

The estimate is 353.04 USD a month, and 310.10 of it is the database. high_availability forces a General Purpose SKU and then runs two of them. That is the decision to look at twice before applying, because it cannot be undone cheaply: high availability can be turned off later, but geo_redundant_backup is fixed when the server is created.

The apply may need running twice. The Key Vault Secrets Officer grant is created in the same apply that writes the first secrets, and Azure RBAC takes a minute or two to propagate, so a second apply after the first fails on a secret write is normal.

A partly finished apply needs no hand cleanup. Terraform records every resource that succeeded, and running plan again asks for exactly the remainder. Read that plan the same way as the first: it should add what is missing and destroy nothing.

Sign-in does not work yet. The OAuth values in the vault are placeholders and the next three steps replace them.

6. Bind the certificate

Terraform has added the hostname and created the certificate. Until something attaches one to the other, the name resolves and the TLS handshake is reset by the peer with no certificate offered at all.

Whether anything has to be that something is currently an open question, so this step checks first and fixes second. Under the older domain.tf the bind below was a person’s job, and the one recorded stand-up of this stack is the evidence: afcpprod-unreachable held Sev0 for ninety five minutes with the certificate issued and nothing serving it. domain.tf has since been rewritten so that the hostname is bound with no certificate and Azure attaches one itself when it issues, asynchronously and outside any apply. If that holds, the command below is a no-op.

Nobody knows yet, and the honest reason is that nobody has applied the new configuration. It reasons from the provider’s documented behaviour rather than from an observed apply, which is a good basis for a configuration change and a poor one for deleting a step whose absence is an outage.

So prove it from outside first, because this is the step whose failure looks like a network problem:

curl -sS -o /dev/null -w 'http=%{http_code} sslverify=%{ssl_verify_result}\n' \
  https://app.antifailure.dev/health

sslverify=0 is a certificate the client trusts: Azure bound it without you.

A connection reset means the binding did not take, and this is the remedy:

CERT_ID=$(az containerapp env certificate list \
  -n afcpprod-env -g af-cp-prod-centralus \
  --query "[?properties.subjectName=='app.antifailure.dev'].id | [0]" -o tsv)

az containerapp hostname bind -n afcpprod-app -g af-cp-prod-centralus \
  --hostname app.antifailure.dev --environment afcpprod-env \
  --certificate "$CERT_ID" --validation-method CNAME

terraform plan stays clean afterwards. The custom domain resource carries ignore_changes on the two fields this command writes, which is the provider’s documented handling for an Azure managed certificate.

7. Confirm the assumptions the alerts are built on

Two numbers were derived rather than read, and both are quiet if wrong.

# The connection alert's denominator. Expect 859 for GP_Standard_D2ds_v4.
az postgres flexible-server parameter show \
  -g af-cp-prod-centralus -s afcpprod-pg -n max_connections \
  --query "{value:value,default:defaultValue}" -o json

# The action group actually delivers. This sends a real notification.
az monitor action-group test-notifications create \
  --action-group afcpprod-pager -g af-cp-prod-centralus \
  --alert-type metricstaticthreshold \
  -a email email-0 "you@example.com" usecommonalertschema

Do the second one. A Status of Succeeded in the result is the proof; anything else is a page that will not arrive.

THE RECEIVER NAME IS NOT FREE TEXT and neither is the alert type. Azure matches email-0 against the receivers the action group already has and refuses ActionOrReceiverNotExistedInActionGroup for a name it does not hold, so it has to be the name the alerting module generates rather than a label of your own. --alert-type metric is rejected as invalid; the accepted value is metricstaticthreshold.

8. Create the production OAuth App

This is your job, in a browser, at https://github.com/settings/developers. Production needs its own, not staging’s.

Field	Value
Application name	Antifailure
Homepage URL	https://app.antifailure.dev
Authorization callback URL	https://app.antifailure.dev/auth/github/callback
Enable Device Flow	unticked
Allow wildcard matching	unticked

The callback has to match github_redirect_uri in production.tfvars character for character. A mismatch fails with an error GitHub shows the user and this application never sees. The field takes more than one: GitHub’s form says you may add up to ten redirect URIs, so a second environment does not need a second OAuth App.

Leave wildcard matching off. The registered callback is exact and nothing needs it. While you are there, untick it on the staging OAuth App too: it is on, and nothing there needs it either.

Leave Device Flow off. af login is this control plane’s own device grant, in web/apps/api/src/auth/device.ts, minting afu_ tokens against /auth/device/code on this server; nothing here calls github.com/login/device. Ticking it adds a way to obtain a GitHub token in this application’s name that nothing in the product would ever use.

Generate a client secret and keep the page open. GitHub shows it once.

9. Create the production GitHub App

Also your job, in a browser, at https://github.com/settings/apps. The webhook secret and the private key are the credentials that let a delivery write rows, so sharing staging’s App would mean a staging compromise writing into production’s tenants. Installation ids also differ per App, and github_installations keys on them.

Field	Value
GitHub App name	Antifailure
Homepage URL	https://app.antifailure.dev
Callback URL	leave empty, sign-in uses the OAuth App
Webhook	Active
Webhook URL	https://app.antifailure.dev/webhooks/github
Webhook secret	generate a long random string and keep it
Where can this be installed	Any account

Repository permissions, and what each one is actually for:

Permission	Access	What uses it
Metadata	Read-only	Mandatory for every App.
Contents	Read and write	Reading the manifest and the workflow file, and the pull request that adds the workflow file to a newly installed repository. The write lands on a branch of its own, antifailure/setup, never on the default branch.
Pull requests	Read and write	The one comment per pull request, and the pull request a masking rule change becomes.
Actions	Read and write	The console’s Create environment, Run agents, Run load and Tear down, and cancelling the run that holds an environment when a pull request closes.
Checks	Read and write	The one check run per commit that a branch protection rule can require.

Organization permissions:

Permission	Access	What uses it
Members	Read-only	Membership sync, which is what stops everybody landing with no tenant.

Grant Actions write at creation even if the console’s controls are not in use yet. It is the one on this list where waiting is worse than granting: widening an existing App’s permissions makes GitHub ask every installation to accept the new grant, so adding it later interrupts every customer, and until somebody accepts, the App declares a permission that no installation holds. Every one of those controls, including starting a workload and tearing an environment down, dispatches a workflow_dispatch run of the customer’s own workflow through dispatchWorkflow in web/apps/api/src/auth/github.ts, and without the permission GitHub refuses with 403 Resource not accessible by integration.

Grant Checks. Without it, a pull request gets the comment and no check run, so no branch protection rule can require Antifailure, and the control plane says which grant is missing in the comment rather than failing quietly.

Subscribe to events: Installation, Installation repositories, Repository, Pull request, Workflow run, Check run, Check suite.

The last five are the pull request lifecycle. Pull request is what opens a check on a commit and closes it when the pull request does. Workflow run binds the check to the Actions run, which is the only route this control plane has into the machine holding the environment. Check run and Check suite are the two Re-run buttons: GitHub sends the first when somebody re-runs one check and the second when they re-run all of them from the checks page, so subscribing to only one leaves the other doing nothing at all. Each is handled in web/apps/api/src/github/lifecycle.ts.

The third Re-run button, the one in the Actions tab, sends neither of those. It starts another attempt of the same workflow run, which arrives as Workflow run, and that attempt then asks for a credential of its own. The control plane reads the attempt number GitHub signed into the run’s identity and reopens the check for a later attempt of the run already checking the commit, so a re-run from either place produces a new check run with a fresh verdict.

Push is still deliberately absent: nothing handles it, and an event nobody consumes is delivery-log noise that makes a real failed delivery harder to find. Member and Membership are absent for a sharper reason: the handler names them and answers handled: false, because membership is resolved at sign-in and reconciled by Sync from GitHub on the Members page. Subscribing to them looks like membership is event driven and it is not.

Adding either of these to an App that already exists

Widening an App’s permissions does not grant them. GitHub raises a request against every existing installation and nothing changes until a person accepts it, so the App’s settings page can read Checks: Read and write while every installation still holds none of it. That is not a hypothetical: it cost most of an hour on Actions: write, where a 403 was read as a code problem for as long as it took somebody to look at the installation rather than at the App.

1. The App’s settings, Permissions and events, Repository permissions, Checks to Read and write, then Save.
2. The same page, Subscribe to events, tick Pull request, Workflow run, Check run and Check suite, then Save. Event subscriptions take effect without anybody accepting anything; only the permission needs step 3.
3. For every account the App is installed on: its Installed GitHub Apps settings, the App, Review request, Accept new permissions.

Contents write is the third such widening, after Actions and Checks, and it is the one the setup pull request needs. Until an installation accepts it, the App can still read the repository and cannot write the workflow file, so the control plane records the refusal rather than retrying it, and the console’s Environments page shows the repository under Getting connected as needing the permission, with the two steps above as the remedy. The new_permissions_accepted delivery that follows the acceptance is what puts the setup back in the queue; nothing has to be restarted.

An installation token minted before step 3 is cached for an hour and carries none of the new grant, so a permission accepted at 00:38 can still be refused at 01:30, and the refusal looks exactly like the permission never having been granted. Restarting the control plane clears it, because those tokens live only in memory and nothing writes them anywhere.

Then, on the App’s page, Generate a private key. GitHub downloads a .pem and never shows it again. Note the numeric App ID at the top of the page.

10. Put the four values in Key Vault

Three of these replace placeholders Terraform seeded; two are ones Terraform deliberately does not own.

Two of them are credentials and go in through the afsecret helper on the Azure page, which takes the value at a prompt rather than as an argument. rotating-secrets.md states the rule for every other credential on this plane: a value passed as --value is in your shell history and in the argument list of a running process, where ps shows it to anybody else on the machine.

The client id is not a credential and stays as an argument. keyvault.tf says so itself, in the comment about what tfsec reports over these three: an OAuth client id is in the address bar of every person who signs in. Putting it behind a hidden prompt would suggest to the next reader that it is the same kind of thing as the two below it.

VAULT=afcpprod-kv-centralus

az keyvault secret set --vault-name "$VAULT" --name github-client-id --value '<oauth client id>'

afsecret github-client-secret
afsecret github-app-webhook-secret

az keyvault secret set --vault-name "$VAULT" --name github-app-private-key --file ~/Downloads/<app>.private-key.pem

The private key goes in as a file. A PEM pasted through a shell loses its newlines, and the application fails to sign a JWT with an error about the key format rather than about how it was pasted.

11. Tell Terraform the App exists, and apply again

Set github_app_id in production.tfvars to the numeric id from step 9, then plan and apply. The plan reads the two secrets you just wrote, and fails if either is missing, which is the check working.

Then read what is actually serving. This is the trap that has caught this project three times. Terraform owns the container app template and continuous deployment owns the traffic, so an apply that adds an environment variable creates a new revision at zero percent and reports success while production keeps serving the old one without the change.

A change to the app’s configuration alone no longer needs this section. Since 2026-09-06 the production job in cd.yml runs deploy/cd/apply-config.sh production after the approval and before deploy.sh. It plans production.tfvars targeted at the container app, applies it only when the plan is an environment or secret reference change and nothing else, and leaves the revision at zero percent for deploy.sh to supersede a minute later with the one that takes traffic. So a variable merged into production.tfvars reaches production on the next tag, through the migration and both health gates, with nobody at a terminal. The job log says what changed by name, or why it refused. Staging gets the same on every merge to main. Everything else in this file, a SKU, a grant, the alerting module, a Key Vault change, is outside that target and still needs the hand apply above; the guard refuses a plan that carries one.

az containerapp ingress traffic show -n afcpprod-app -g af-cp-prod-centralus -o table
az containerapp revision list -n afcpprod-app -g af-cp-prod-centralus \
  --query "[?properties.active].{rev:name,created:properties.createdTime}" -o table

If the newest revision is not the one with the weight, move it:

az containerapp ingress traffic set -n afcpprod-app -g af-cp-prod-centralus \
  --revision-weight <newest-revision>=100

Read that from Azure and not from Terraform. The stored state file records the traffic weight from before the last deploy and ignore_changes deliberately keeps it there, so it is stale by design and says nothing about what is serving. An empty plan is not an answer either, because the attribute that would say so is the ignored one. See the revision mode trap.

12. Install the App on the organization

On the App’s page, Install App, and choose the account and repositories. Nothing has a tenant until an installation exists.

Installing is not the same as being installed, and the difference is a webhook this control plane may have refused. Installing sends one installation delivery, once. GitHub does not retry a webhook. So if the App was installed before step 11, which is the order the App’s own setup page encourages, because Install App is on the page you are already looking at, then the delivery arrived at a control plane whose AF_GITHUB_APP_WEBHOOK_SECRET was unset, was answered 503, and is gone. github_installations stays empty, every sign-in lands with no organization, and nothing anywhere says why.

So check it. The App’s Advanced tab lists every delivery with the status code this control plane returned, and each row has a Redeliver button. Use that tab: gh api /app/hook/deliveries does not work here, because the deliveries endpoint authenticates as the App and gh holds a user token. The API route needs a JWT signed with the App’s private key, which is the same key you put in the vault in step 10.

If the installation row is not 200, redeliver it. The response body is the check that matters, and a successful one names the installation:

{"event":"installation","action":"created","handled":true,
 "detail":"installation 157834739 for antifailure, 1 repositories"}

One trap if you script this instead. Delivery ids are past the range a double holds exactly, 3839993231035072512 being a real one, so a JSON parser backed by doubles rounds the last digits and JavaScript’s JSON.parse turns that id into …072500. A redelivery aimed at the rounded id is a 404 on a delivery that never existed. Take the id out of the raw body as text.

13. Let continuous deployment reach production

The federated credential already exists. Do not create it. Checked rather than assumed: af-infra-ci carries eight, including github-env-production and github-env-production-immutable, which are the two spellings of repo:<owner>/<repo>:environment:production. Both are registered because GitHub has moved to immutable OIDC subjects carrying numeric organisation and repository ids, and an application takes twenty credentials, so keeping both means a change in either direction does not break the job.

Confirm rather than trust this page:

APP_ID=$(az ad app list --display-name af-infra-ci --query "[0].id" -o tsv)
az ad app federated-credential list --id "$APP_ID" \
  --query "[?contains(subject,'environment:production')].{name:name,subject:subject}" -o table

What is missing is the role assignment, because the production group does not exist until step 5 and a grant cannot precede its scope. The identity needs on the production group what it already has on staging’s: Contributor, scoped to that group and nothing wider.

Terraform owns it. Do not create it by hand. This page used to print an az role assignment create here and that instruction outlived the code that replaced it, which is worse than either alone: a grant made by hand is absent from the stack’s state, cannot survive a rebuild, and reads to the next person as a resource Terraform does not manage. The grant is azurerm_role_assignment.cd_deploys_the_group in stacks/control-plane/ci.tf, and it is switched on by cd_principal_id in production.tfvars, which is already set. Step 5 creates it along with everything else.

Confirm it after the apply, rather than trusting this page:

az role assignment list \
  --assignee "$(az ad app list --display-name af-infra-ci --query '[0].appId' -o tsv)" \
  --scope "$(az group show -n af-cp-prod-centralus --query id -o tsv)" \
  --query "[].roleDefinitionName" -o tsv

If that prints nothing, cd.yml’s production job fails at its first az containerapp call and continuous deployment cannot reach production at all.

14. Set the approval rule on the production environment

In repository settings, Environments, production: add required reviewers. The cd.yml job does not start until somebody clicks it, and the reviewer list lives there rather than in an if: a pull request can edit in the same commit that deploys.

15. Release

Push a v* tag. Continuous deployment builds once, deploys to staging, waits for the approval, and then promotes the same image digest staging tested. The production job asks Azure whether the app exists before doing anything, so it refuses cleanly if any of the above was skipped.

After the first release

• Watch the availability alert clear rather than assuming it did. It is severity 0 and it fires on two failed probe locations.
• Run the backup drill and write down the number it prints. That number is your recovery time objective and nothing else is. The operations page has the command.
• The runbooks are the pages the alerts link to. Read the index once now, while nothing is broken.

Turning billing on

Billing is off on a control plane that has never been told about Stripe. Every route that would charge answers PRECONDITION_FAILED naming the settings it needs. What follows turns it on; run the sections in the order they are written.

Three settings, and two of them are credentials. web/apps/api/src/billing/plans.ts requires exactly these:

Setting	Secret	Where it comes from
AF_STRIPE_SECRET_KEY	yes, Key Vault	Stripe, Developers, API keys
AF_STRIPE_WEBHOOK_SECRET	yes, Key Vault	shown once, when you create the webhook endpoint
AF_STRIPE_PRICE_TEAM	no	stripe_price_team in production.tfvars

Two of three is worse than none. A partial configuration is reported as a refusal, not as a partial success: the process prints billing is OFF and partially configured with the missing names in it and takes no money at all. That is deliberate, because the setting people forget is the webhook secret, and an installation missing only that one appears to work right up until the first customer pays and never gets what they bought.

There is no AF_STRIPE_PRICE_ENTERPRISE and there is not meant to be one. Enterprise is agreed with a person, so no Stripe price exists behind it. Checkout refuses that plan by name and points at the contact route.

First, create the webhook endpoint at Stripe

Your job, in a browser, at https://dashboard.stripe.com/webhooks. This step is first because AF_STRIPE_WEBHOOK_SECRET does not exist until you do it: Stripe generates the signing secret when the endpoint is created and shows it once.

Field	Value
Endpoint URL	https://app.antifailure.dev/webhooks/stripe
Listen to	Events on your account
API version	your account default

Select exactly these nine events, which are the ones HANDLED_EVENTS in web/apps/api/src/billing/webhook.ts acts on:

customer.subscription.created, customer.subscription.updated, customer.subscription.deleted, invoice.paid, invoice.payment_failed, invoice.finalized, payment_method.attached, payment_method.detached, checkout.session.completed.

Subscribing to more is harmless and subscribing to fewer is not. An event this control plane does not act on is acknowledged and not recorded, so a wider selection costs a 200 and nothing else. A narrower one loses an entitlement.

Do this in test mode first, against a control plane you can afford to be wrong about. Test mode has its own endpoint, its own signing secret, its own keys and its own prices, and nothing crosses between the two.

Then put the two credentials in Key Vault

The vault name is afcpprod-kv-centralus for production. Use the afsecret helper on the Azure page, which takes the value at a prompt rather than as an argument, writes it with no trailing newline, and removes the file afterwards. A signing secret with a trailing newline fails every signature and the endpoint answers 401 to every delivery Stripe makes.

Confirm both are there before going on. This prints names, never values:

az keyvault secret list --vault-name afcpprod-kv-centralus \
  --query "[?starts_with(name, 'stripe-')].name" -o tsv

Two names, or stop here.

Then set the price, and only then apply

stripe_price_team in production.tfvars is the switch. Setting it makes the container app reference both vault secrets by their versionless ids.

The plan cannot tell you the secrets are missing. keyvault.tf addresses them by constructed id rather than reading them, so a plan is green whether or not the secrets exist, Azure discovers a missing one while resolving references during deployment, and the revision fails to start on a control plane that was serving a moment earlier. Putting the credentials in the vault is not reorderable.

Apply, then shift traffic the way every other change to this app is shifted: the app runs in Multiple revision mode, so the apply creates a revision at zero traffic. Probe it at zero, then shift.

Then prove it, on the running control plane

A route that answers 200 is not proof that a plan changed. Four checks, in order, each of which can only pass if the one before it did.

The endpoint stops refusing. Before, this is a 503 saying this control plane is not configured to take payments; after, it is a 401, because the request is now being checked against a signing secret rather than turned away:

curl -sS -X POST https://app.antifailure.dev/webhooks/stripe \
  -H 'content-type: application/json' --data '{}' -w '\n%{http_code}\n'

A 503 here means one of the three settings did not arrive. A 401 means all three did, and that the process is verifying signatures.

Then Send test webhook from the endpoint’s page in the Stripe dashboard. A 200 proves the signing secret is byte for byte the one Stripe holds, which is the half a 401 above cannot distinguish from a wrong secret.

Then buy something in test mode, with Stripe’s 4242 4242 4242 4242 card, and watch the organization’s plan change. Not the checkout page opening: the plan.

Then ask the product for the thing the plan was withholding. Create an environment that the free plan’s limit of three refused before the purchase. That is the only check that cannot be satisfied by a payment path that is connected to nothing.

Turning on the enterprise edition

The hosted control plane runs ghcr.io/antifailure/control-plane-enterprise, the image whose entry point mounts single sign-on and directory provisioning and starts the audit stream forwarder. cd.yml builds and deploys that image to staging on every merge and to production on every tag. The community image is still built and published for self-hosted installations, and nothing here changes it.

That image will not start on an app that has not been given its edition. Without AF_EE_SSO_KEY the process exits before it listens, whatever the licence says, and a licence with no AF_ORG or no trusted key stops it at start-up with exit status 2. This is a one-time procedure per environment, and it runs before the first release that deploys the enterprise image there. Run the sections in the order written.

Three settings in the tfvars file, and two secrets in the vault.

Setting	Where it lives	Who creates it
enterprise_edition = true	the environment’s tfvars	a person, in a pull request
license_org, license_public_keys	the environment’s tfvars	a person, in a pull request; both are public
ee-sso-key	the environment’s vault	Terraform, in one targeted hand apply
license-key	the environment’s vault	a person, from tools/licensegen

The order is forced by two gates, not chosen. tools/configguard refuses a configuration apply that creates anything other than an environment or secret reference change, and ee-sso-key is a vault secret Terraform creates, so the apply cd.yml runs cannot create it and refuses the whole release instead. And deploy/cd/edition-check.sh configured runs before every deploy and refuses one whose app template is missing any of the four variables, leaving the app serving what it served. Both refusals leave production untouched, and both spend a release approval to tell you something this page already says.

Staging was switched on this way when the enterprise image first shipped. The commands below are production’s, with afcpprod-kv-centralus, afcpprod-app and af-cp-prod-centralus.

First, issue the licence and put it in the vault

The hosted licence is signed by license-signing-key-hosted-2026-09, the one signing key both hosted environments trust. Its public half is license_public_keys in both tfvars files. Read where the key lives, and who can read it, in issuing a license before you use it.

The private key goes from the vault into the environment of one command and the licence goes from that command into a file only you can read, then into the vault. Neither is ever printed.

umask 077
dir="$(mktemp -d)"
cat > "$dir/request.json" <<'JSON'
{
  "org": "antifailure",
  "plan": "enterprise",
  "features": ["audit_stream", "rbac", "scim", "sso", "support_access"],
  "seats": 0,
  "months": 12
}
JSON

AF_LICENSE_SIGNING_KEY="$(az keyvault secret show --vault-name afcp-kv-centralus \
    --name license-signing-key-hosted-2026-09 --query value -o tsv)" \
  go run ./tools/licensegen issue -request "$dir/request.json" \
    -key-id hosted-2026-09 -id hosted-production-2026-09 \
  | tr -d '\n' > "$dir/licence"

az keyvault secret set --vault-name afcpprod-kv-centralus --name license-key \
  --file "$dir/licence" --output none
rm -P "$dir/licence" 2> /dev/null || rm -f "$dir/licence"
rm -rf "$dir"

Read the receipt on standard error before going on. Its second line names a public key. It must be exactly the value after hosted-2026-09= in production.tfvars, or every start refuses the licence as signed by a key this installation does not trust.

org is antifailure because that is license_org, and the two are compared at start-up. It names the installation, not a customer: each customer organization on the plane is still gated by its own plan. seats is zero, which is unlimited, because the hosted plane counts seats per organization by plan and a licence limit here would cap every customer at once. Twelve months means the licence expires a year from the moment it is signed and then runs on its fourteen day grace; af license status and the start-up line both say how many days remain, and a renewal is this section again with a new -id.

Confirm it is there, as a name and a length, never a value:

az keyvault secret show --vault-name afcpprod-kv-centralus --name license-key \
  --query value -o tsv | tr -d '\n' | wc -c

Staging’s, issued with exactly these commands on 2026-09-12, is 432 characters. A length far from that is a failed signing or a stray byte written into the vault, and the next start will refuse it.

Then generate the sealing key, with one targeted apply

ee-sso-key is owned by Terraform for the reason provider-key-secret is: no person ever holds it, and nothing regenerates it, because a new key cannot open anything the old one sealed. With enterprise_edition = true merged into production.tfvars, from a checkout of that commit:

cd infra/terraform/stacks/control-plane
terraform init -reconfigure -backend-config=backend.production.hcl
export TF_VAR_subscription_id="$(az account show --query id -o tsv)"
export TF_VAR_github_client_id=seeded-once-not-read-here
export TF_VAR_github_client_secret=seeded-once-not-read-here
terraform plan -var-file=production.tfvars -out=edition.tfplan \
  -target='module.control_plane.azurerm_key_vault_secret.owned["ee-sso-key"]'
terraform apply edition.tfplan

The two GitHub values are placeholders on purpose: the seeded secrets ignore their value after the first apply, and this target does not reach them.

Read the plan before applying it. It must say exactly 2 to add, 0 to change, 0 to destroy: random_bytes.ee_sso_key[0] and azurerm_key_vault_secret.owned["ee-sso-key"]. Anything else is a change to a resource this procedure has no business moving, and the answer is to stop, not to apply.

Then both names, never values:

az keyvault secret list --vault-name afcpprod-kv-centralus \
  --query "[?name=='ee-sso-key' || name=='license-key'].name" -o tsv

Two names, or stop here.

Then release

Push the tag. The production job’s configuration apply now plans an environment and secret reference change and nothing else, so configguard accepts it and names the four variables it added. edition-check.sh configured finds all four in the template, deploy.sh moves the image and the traffic, and edition-check.sh serving asks the public origin for the provisioning discovery document, which must answer 200, and for the single sign-on discovery route with no email address, which must answer 400 asking for one. Both routes are on the single sign-on and SCIM pages. A 404 from either is the community image. A 402 is the enterprise image with a licence that does not permit that feature, and the body names the licence state.

Then move the image defaults, in a commit after the tag

The container app’s image belongs to deploy.sh, but the maintenance job reads image_repository and image_tag from infra/terraform/stacks/control-plane/variables.tf with no ignore_changes, so the next hand apply puts that image back on it. Once the tag exists, change the repository to ghcr.io/antifailure/control-plane-enterprise and the tag to the release in one commit. Not before, and not inside the tag’s own commit: tools/tagsync refuses an image_tag bump in the tagged commit, and refuses a repository whose Dockerfile did not exist in the tree that tag names, because the registry has no such image and the maintenance job would pull a manifest that is not there.

Then prove it, on the running control plane

The deploy job has already asked for both routes. Ask again yourself, and read what the process said it decided, because a route that answers is not the same claim as a licence that says what you issued:

deploy/cd/edition-check.sh serving https://app.antifailure.dev 1 0
az containerapp logs show -n afcpprod-app -g af-cp-prod-centralus --tail 200 \
  | grep -E 'license|licence|mounted|audit stream'

The check names both routes and says they are mounted and licensed, then the log shows a start-up line naming the licence as active for antifailure with its features and expiry, the two extensions mounted, and the audit stream line. With no AF_AUDIT_STREAM_SINK set, that line says the audit log is written and not forwarded, which is correct for a plane where each organization chooses its own destination.