Masking

Masking replaces every value that identifies a person with a synthetic one, while keeping everything a test depends on: shapes, lengths, formats, joins, distributions, and uniqueness.

That second half is the whole difficulty. Nulling every string is easy and gives you an environment where nothing renders, no form validates, and no join returns a row. A masked database has to still behave like the one it came from.

Where it runs

On a golden candidate, never on a source.

AF-MSK-005 Masking is only permitted on a golden candidate, and
postgres://prod/app is a source database.
  Next: Run masking against a golden candidate; the engine never rewrites a
  source.

The source is read once, with pg_dump, and never written to. There is no flag that changes this.

The rules

masking.yaml

rules:
  - table: users
    column: email
    transform: email
    why: "customer addresses"

  - table: "*"
    column: "*_id"
    type: uuid
    transform: uuid_remap
    link: entity
    why: "keeps foreign keys joinable after remapping"

table and column accept *. type matches the type name, written in the Postgres vocabulary whichever store the column is in: see more than one store. why is one sentence, printed by af mask plan beside the column it applies to, so a decision made months ago is readable when somebody questions it.

link is the one that catches people

Two columns joined by a foreign key must mask to the same value, or the join returns nothing. link groups them:

  - table: users
    column: id
    transform: uuid_remap
    link: user
  - table: orders
    column: user_id
    transform: uuid_remap
    link: user

Without the link, users.id and orders.user_id get different new UUIDs, every order becomes an orphan, and the environment looks like a customer base with no orders. Nothing errors. That is why it is worth stating explicitly.

More than one store

An environment can hold more than one datastore, and the same person is usually in several of them: a Postgres holding accounts and a ClickHouse holding the events about them, joined on an identifier that is in both.

One identity has to mask to one person across every store. If it does not, a join across the two returns nothing or returns somebody else, every report built on it is plausible, and nothing anywhere says so. That is worse than a store nobody copied at all, because an empty store is visible within a minute of opening a chart.

It is one masking.yaml for every store, and the type in a rule is written in the Postgres vocabulary whatever the store is. Each engine’s own type names are mapped onto the Postgres ones before a rule is matched, so type: text means Postgres text and ClickHouse String and nobody writes the rule twice. A rules file per engine would be a rules file that goes stale for one engine and not the other, and the failure mode of that is a column masked in one store and real in the next.

Two refusals follow from the same principle:

• A store whose engine this build has no dialect for is refused when the plan is made. Guessing at Postgres would mean matching a rule against a vocabulary the store does not have, so nothing would match, so every column would fall through to the branch that says nobody decided. A column that looks classified and was not is the failure this whole page is about.
• A ClickHouse table with no sorting key is refused for the same reason a half masked table is never started. ClickHouse has no physical row identifier, so there is no statement that means one row. Postgres always has ctid, so the refusal is the engine’s rather than a new rule about keys.

The transforms themselves never needed a store. Every one is a pure function of the project key, the column identity, and the input value, computed on the machine running the refresh and never in the database, so what a value masks to has never depended on which store it came out of. What did depend on the store was the classifier deciding what to do with a column, and that is what the dialect settles.

The check that says the two stores agree

The guarantee is checked rather than argued, and you run the check on your own stores rather than reading about ours. Give each datastore the name of the variable holding a read only connection string:

database:
  source_url_env: PRODUCTION_DATABASE_URL

datastores:
  - name: events
    engine: clickhouse
    stance: golden
    source_url_env: CLICKHOUSE_URL

Then:

af mask crossstore

It takes both stores’ plans, finds every identifier that appears in both, masks probe values through each side, and reports the share that come out identical:

join keys verified identical across primary and events: 4 of 4 (100.0%)

It reads catalogs and no rows. The probe values are its own, so what it needs from a store is the schema, which is why it is safe to point at production. The report says how many tables and columns it read and that it read no rows, as a field rather than as a promise on this page.

That is enforced by a test rather than by intent. A live test runs the check against a real ClickHouse, then reads the server’s OWN system.query_log back and fails if any statement the check sent selected from a data table. A sentence saying no rows are read is something anybody can write; a query log the server keeps is something that can contradict it, and if it ever does, the rows_read of zero in the report is a lie and the build says so.

A table the reader deliberately left out is named with the reason, because a share of the join keys it could see is a true answer to a smaller question when half a schema was dropped in silence. A ClickHouse view has no rows of its own and a Distributed engine is a pointer at another server, so neither is a store whose masking can be compared. Both are left out of the comparison and named in the report, rather than dropped in silence.

Every store it could not read is named with the reason, and a store that names no source_url_env is named as never read at all. A run that reached one store says it proved nothing rather than reporting a hundred percent of one, and that answer carries a different exit code from a real disagreement: one is a statement about your data and the other is a statement about what could be reached.

The same question is the cross_store question of the inspect_data_masking tool, where it answers PASS, FAIL or INCONCLUSIVE. Where two stores are present it is also a line in the component inventory, and until something compares them that line reads unmeasured rather than passed.

Anything below 100 percent is a bug, and there are three ways to get there:

• one side is masked and the other is copied unchanged, which is a leak as well as a broken join
• the two sides are masked with different transforms
• the two sides are masked under different links, so each derives its own subkey and one input produces two different outputs

There is deliberately no way to mark a pair exempt. Two columns with one name in two stores that genuinely mean different things is a real thing to look at, and the cost of looking at it is a rule; the cost of silencing it is a twin that is wrong in a way nobody can see. A report that found nothing to compare is not a pass either.

The commonest thing it finds is a blob that is jsonb on one side and a String holding JSON on the other. Nothing leaks, and the two stores still hold different values for one field. A rule settles it:

  - table: "*"
    column: properties
    type: text
    transform: empty_json
    why: "the analytics store keeps this JSON in a String"

What is not built yet

The dialect boundary is the classification, the statements and the verification scan. Nothing yet refreshes a golden for a second store or branches one: there is no ClickHouse provider, and datastores entries other than primary are reported with their declared stance rather than measured. af mask crossstore opens a connection to a second store to READ ITS CATALOG and nothing else; a ClickHouse is read over its HTTP interface, and a URL naming the native port is refused with the HTTP one in the message rather than attempted. Said here rather than left to be discovered, because a boundary that looks complete from outside is how somebody ends up trusting one.

Writing the rules from the schema

af mask init      # reads the source database, writes masking.yaml

af mask init connects to the source the manifest names, reads the catalog, and runs the classifier over it. It writes one rule per column that carries personal data, restating the default that matched with its why, and one explicit rule per column the classifier could not place, so the file records a decision for every column rather than leaving the unplaced ones to the inconvenient default. af mask plan on the result reports zero problems and zero unmatched columns, which is the point: the first plan you read is one where every row is a choice to confirm rather than a gap to fill.

It refuses to overwrite a masking.yaml that exists. Pass --force to replace one, and read the diff, because a rule you edited by hand is what the rewrite would lose. af init runs the same code when the source resolves at init time, and says either that the rules were written from N tables or that they were not written because there is no source yet.

Planning before applying

af mask plan      # every column, the rule that matched, and why
af mask preview   # before and after, on a sample, values redacted
af mask apply     # run it against a candidate
af mask verify    # scan the result

af mask plan is the one to read. It lists every column in the schema, which rule matched it, and what will happen. A column with no rule is shown as such, which is how you find the notes field nobody thought about.

Its first lines are the two counts that matter:

Masking plan
  Read from the source named by AF_SOURCE_DATABASE_URL.
  231 columns across 55 tables, about 6111 rows.
  128 columns have no rule, and 128 of those are copied unchanged.

Two different things share “no rule”. Most such columns are emptied by the fail closed default, which is a question with a safe answer already in place. The rest are copied unchanged: a NOT NULL text column, a bytea, an enum, an array, anything the default has no way to empty. Those hold exactly what production holds, and that count is the one to read first. It is printed at the top because the list it summarises is printed at the bottom, after every assignment, and on a real schema that is several hundred lines down.

The same count travels. af mask apply and af golden refresh print “N columns copied unchanged with no rule” beside their own success line, af mask verify prints it beside its verdict, the golden’s attestation records the count and the names, and af golden list shows it in a column called NO RULE, so a golden made from a rules file with a gap in it says so wherever the golden is looked at.

Columns with no rule

AF-MSK-008 The columns orders.notes, tickets.body hold free text and have no
masking rule.
  Next: Give each column a rule, or allowlist it explicitly if it is known to
  hold no personal data.

Unclassified free text defaults to nullify, because a column nobody has confirmed is safe is a column that might hold anything a customer typed. That default is deliberately inconvenient: it makes the page render wrong, which makes somebody look.

To keep the shape, give it free_text. To state it was reviewed and is safe, give it preserve and a why.

A rule that names nothing

AF-MSK-003 The masking rule for users.emial names a column that does not exist
in the schema.
  Next: Remove the rule or correct the name; 'af mask plan' lists the columns
  it found.

A typo in a rule is a column with no masking and no warning, so a rule that matches nothing is an error rather than a shrug. Wildcards are exempt: column: "*_id" matching nothing in a small schema is normal.

Third party identifiers

A Stripe customer id, a subscription id, an invoice id: none of these is a secret, and every one of them is a live pointer into a real account. Stripe issues it once and it never changes, so anybody who has seen it in an invoice email or the Stripe dashboard can say which real customer a masked row belongs to, and it works the same in every environment because the value is the same in every environment.

They are NOT NULL text in almost every schema, so the default cannot empty them, and they are copied unchanged until a rule names them. prefixed_id keeps the prefix and replaces the body with a keyed hash of the same length:

  - table: subscriptions
    column: stripe_customer_id
    transform: prefixed_id
    link: stripe
    why: "a live pointer into a real Stripe account"

The billing code still recognises cus_ as a customer, the unique constraint holds, and with one link across every table that carries the id the same customer maps to the same fake customer in all of them. The masked body is lowercase hex, which the verification scan’s provider identifier detector does not report, so the scan can still catch a real one.

Columns the scan cannot read

A bytea column holds whatever was written into it, and the verification scan cannot pattern match a sealed blob. It decodes each value as UTF-8 where it decodes and lists the column as not readable where it does not. A column the scan cannot read is masked by its rule or by nothing, so give every one a rule: nullify where the column allows it, hash_hex where it does not.

  - table: provider_keys
    column: ciphertext
    transform: hash_hex
    why: "the customer's sealed API key; the api refuses a body its tag does not authenticate"

Read how the application treats an unreadable value before choosing. A decryption that fails closed with a typed error is what you want on the copy; a crash is not.

What masking does not decide

Whether the result is safe. That is verification, which runs afterwards, scans for anything that still looks like a person, and refuses to publish if it finds something. The rules are a claim; the scan is the check.

Related: transforms, goldens.