The differential oracle

A test says whether the application does what you told it to. The oracle says what this change did that the last version did not, which is a different question and usually the one being asked in a review.

It brings a second environment up from a baseline revision, branches the same golden for both so they start from identical rows, sends both the same requests in the same order, and reports every difference in what came back and in what ended up in the database.

af oracle
af oracle --baseline v2.4.0
af oracle --keep --report oracle.md

What is compared, and what is not

Responses and database contents. Not events, not outbound effects, not traces, not query plans.

That is a decision rather than an omission. Two comparisons done completely are worth more than six done shallowly, because the first check that reports a difference which is not one is the last check anybody looks at. When the other four arrive they will arrive finished.

What is compared:

Status code	Exactly, and by class. A status that falls into an error class outranks one that moves inside its class.
Response headers	Every header except a default list of the ones no two runs agree on. The list is printed on every run.
JSON bodies	Structurally, by path. Key order in the document is not a difference; a field that appeared, disappeared, changed type or changed value is.
Other bodies	By content. The report says the two differ and where, and does not attempt a text diff.
Database contents	Every table, row by row, matched on the primary key, with each column compared.
Table structure	Columns added, dropped, or retyped between the two sides.

The baseline

oracle.baseline decides which revision the comparison is against, and the two values answer different questions.

merge_base, the default, is the commit this branch and the base branch share. It answers “what does this branch change”, and it does not move when somebody else lands a commit on the base branch halfway through a review.

ref is a revision named outright: a branch, a tag, or a commit. It answers “what changes when this ships”, which is what a release gate wants.

There is no value for “the revision currently deployed”, because the engine cannot know what that is. A deployment pipeline does, and it passes the commit:

af oracle --baseline "$DEPLOYED_SHA"

With no base_ref set, the comparison tries origin/HEAD, then origin/main, then origin/master, then main, then master, and the report says which one it used.

Two environments, one golden

The two versions cannot share an environment. They want the same ports, the same service names and the same database.

They do share a golden. The candidate comes up first and the baseline is pinned to whatever golden version the candidate branched, so a scheduled refresh landing between the two cannot separate them. That matters more than it sounds: if the two sides start from different rows, every row in the report is noise and the comparison says nothing.

Only the images are built from the baseline checkout. The manifest, the egress policy, the personas, the ports and the secrets all come from the candidate’s manifest. If the baseline’s own manifest were used, a manifest change in the pull request would move the application and the harness at once, and no difference in the report could be attributed to either.

The candidate environment is left running whether or not af oracle brought it up. The baseline is torn down unless --keep says otherwise.

The probes

Both versions have to receive the same bytes in the same order, so the plan is written down rather than discovered:

oracle:
  probes:
    - name: list-customers
      method: GET
      path: /customers
    - name: place-an-order
      method: POST
      path: /orders
      headers:
        content-type: application/json
      body: '{"customer_id": 1, "total_cents": 2599}'

Each probe goes to the baseline and then immediately to the candidate, rather than the whole plan to one side and then the whole plan to the other. Any value that comes from the clock is much more likely to agree when the two requests are milliseconds apart, and a probe that depends on an earlier probe’s write sees the same state on both sides at the same point in the sequence.

Requests are sent one at a time. Concurrency would make the order of the two databases’ writes depend on scheduling, and then the identifier a row got would depend on scheduling too.

The agents that drive a workflow are not used here. They decide their next step from what is on the screen, so two runs of one workflow send two different request sequences, and a comparison of those compares the agent with itself.

Non-determinism

A byte comparison of two responses reports a different Date, a different session cookie, a different request identifier and a different generated timestamp on every single request. So values are normalised before they are compared, and every normaliser is narrow on purpose.

Source	What happens
Clocks	Two strings that both parse as a timestamp and are within an hour of each other are equal. Further apart, they are reported. One side a timestamp and the other not is reported.
Random identifiers	Two strings that are both UUIDs are equal.
Sequence identifiers	Compared exactly, deliberately. See below.
Floating point	Numbers are equal within a relative tolerance of 1e-9, so representation noise is not news.
Session cookies, request ids	Set-Cookie, ETag, Date, X-Request-Id and nine others are not compared. The full list is printed on every run.
Ordering of writes	Requests are sent one at a time, and rows are matched on the primary key, so storage order is never a difference.

The hour is configurable, and every run says how wide a gap the timestamp normaliser actually absorbed. A gap of four milliseconds is the harness; a gap of fifty minutes is worth a look.

What is not normalised is as considered as what is.

Sequence identifiers are compared exactly. Both databases branch one golden and receive the same requests in the same order, so the sequences have to agree. A sequence at 41 on one side and 42 on the other means the candidate wrote a row the baseline did not, which is the most useful thing this comparison can tell anybody. Normalising identifiers away would have thrown it out.

A numeric epoch is compared exactly. Deciding that a number is a clock from the name of the field it sits under would silently ignore an expiry that moved by a day. When a number under a name like expires_at differs, the report says so and prints the line that would ignore it.

An opaque token that is neither a UUID nor a timestamp is compared exactly. There is no shape to recognise, and this is not a place to guess. Ignore it by path.

Everything the comparison declined to look at is printed, defaults included, assembled while comparing rather than described in a document. An oracle that silently ignores a field is worse than one that reports it, because the field it ignored is where the bug was.

What counts as a difference worth reporting

Findings are ranked, and the ranking is directional. A candidate that stops returning a field, stops writing a row, or turns a served request into an error has lost something the baseline had, and that is rarely intended. A candidate that returns an extra field or writes an extra row is what a feature branch does all day.

Critical. A request the baseline served and the candidate did not answer at all. A status that fell into an error class. A row the baseline wrote and the candidate did not. A body declared JSON that no longer parses.

Major. A status that moved inside its class. A field the baseline returned and the candidate does not. A value that changed JSON type. An array that lost elements. A media type that changed. A row whose columns disagree. A table or a column the baseline has and the candidate does not.

Minor. A field or a row the candidate added. A scalar value that changed. An array reordered with the same members. A compared header that changed. A status that left an error class.

oracle.fail_on decides which of those fails the command, and defaults to critical. A pull request exists to change behaviour, so failing on any difference at all would fail every branch and teach everybody to pass the flag that turns it off.

Database contents

The two branches are compared by their contents rather than by the statements that produced them.

Logical decoding needs a replication slot and an output plugin installed in the database, and audit triggers need schema changes on every table in a database that is supposed to have production’s shape. Both also answer a question nobody asked, which is which statements ran. What a review needs to know is what a row holds.

Two snapshots are taken on each side, one before any request and one after. A row that already differed before either version served a request is the migrations’ doing; a row that differs only afterwards is the application’s. The report labels each finding with which it was.

Tables are read inside a read only repeatable read transaction, so Postgres refuses a write rather than this code promising not to make one, and every table is read at one instant.

A table with more rows than oracle.database.max_rows, ten thousand by default, is reported as not compared, with its approximate size. It is never silently skipped: a report that omits a table reads exactly like a report that found nothing wrong in it.

A table with no primary key has its rows matched on their whole content, so an update reads as one row removed and one row added. Without a key there is no fact about which row on one side corresponds to which row on the other.

Ignoring a field

oracle.ignore.fields takes the subset of JSONPath people actually write:

oracle:
  ignore:
    headers: [x-served-by]
    fields:
      - $.payment_intent
      - $.orders[*].reference
      - $..updated_at

$.field selects one field, $.list[0] one element, $.list[*] every element, $..name that name at any depth, and $.object.* every field of one object. A pattern that does not parse is refused when the manifest is validated, rather than matching nothing quietly.

A path applies to a response body and to a table row alike. A row’s path is $.<column>, so $..updated_at written once covers the response field and the column behind it.

Paths in the report are written in the same syntax, so one can be copied out of a report and pasted into the manifest.

Limits

These are real and are not going to be discovered by surprise.

• An insert and a delete inside one request are invisible, because the comparison is of contents and the net effect is nothing.
• A background worker that writes a different number of rows on two runs will report a difference that is not the change. Exclude its tables.
• Non-JSON bodies are compared by content, not by structure. A probe pointed at an HTML page will report a difference for a CSRF token. Point probes at endpoints that return JSON.
• A new service in the candidate’s manifest fails the baseline build, since the baseline checkout has no source for it. That is a change the comparison cannot make, and it says so rather than comparing what is left.
• The comparison costs a second environment. On a copy-on-write database provider the second branch is nearly free and the second build is usually a cache hit; the containers are not.

Configuration

oracle:
  enabled: true
  baseline: merge_base        # or ref
  base_ref: origin/main
  fail_on: critical           # none, minor, major, or critical
  compare_timestamps: false   # true compares timestamp strings exactly
  compare_uuids: false        # true compares UUIDs exactly
  probes:
    - name: list-customers
      path: /customers
  ignore:
    headers: []
    fields: []
  database:
    enabled: true
    tables: []                # empty compares every table
    exclude: []
    max_rows: 10000

The block is absent by default. The comparison doubles the environments a run costs and it needs a probe plan somebody wrote, so it does not happen unless a manifest asks for it. A block that is present with enabled: false is a different answer from no block at all: it is a probe plan somebody kept and a check they turned off, and af oracle says so and exits zero.