Restore Drills in 2026: You Can't Automate the Declaration, Only the Comparison

The previous post in this little series — Backup Verification in 2026: Exit Code 0 Is Not a Backup — grew out of a comment thread under a dev.to article by dafili, "My Backup Exited 0 for Three Weeks and Nothing Paged Me". The conversation did not stop at publication. Dafili corrected one claim in that post (the correction is in, and credited), and then pushed the whole idea one level deeper — to a place the four-level ladder does not reach on its own.

His point, in his words: "You cannot automate the declaration, only the comparison."

This post is about what that sentence means, because once you hear it, a hole opens up in the middle of most restore drills — including the tidy ones from the ladder post. The ladder tells you *how* to check a backup. It never tells you what "correct" means. And if the definition of correct comes from the same system you are checking, your drill can pass beautifully while verifying nothing at all.

> TL;DR: A restore drill is only as honest as the expectations it compares against. If those expectations are derived from the backup or the system being verified — row counts read from the dump itself, file lists generated by the backup tool — the check is circular: it catches corruption, never wrongness. The fix is a fingerprint: a short, hand-written declaration of what a correct restore must contain (an expected file list, one file's sha256, a row count, one config value). A human re-declares it quarterly; automation compares against it nightly and shouts on drift. And the drill itself gets monitored with a days_since_last_successful_manual_restore gauge that pages at about 120 days — because an unmonitored drill quietly stops happening.

The four levels from the previous post — the artifact exists, it is intact, it can be read, it has actually been restored — still stand. Level 4 is where trust lives: restore into a scratch environment and verify the result like a user would.

But look closely at how a typical level-4 drill "verifies." It restores the dump, then runs queries against the restored database: count the rows in users, check that orders has recent entries, confirm the application boots. Where do the expected values for those checks come from? Usually from the restored data itself, or from the same monitoring that watched the original system. The drill asks the backup: *do you agree with yourself?* The backup always says yes.

That is how a drill passes on data that is perfectly intact and completely wrong. Dafili made this distinction in the same thread: an integrity check catches media or repository corruption, but only a restore catches data that is wrong — and even then, only if the restore is compared against something that did not come from the restore. An empty mount backed up faithfully, checksummed faithfully, restored faithfully. A dump of the wrong database, in perfect condition. Every level green. Nothing recovered that you actually wanted.

He explored the same trap from another angle in a follow-up article, "Three checks that look like they passed but validated nothing": checks that execute correctly, return success, and prove nothing, because they validate a different property than the one you care about. A restore drill with self-derived expectations is exactly that shape of check — bigger, more expensive, and just as capable of validating nothing.

Dafili's answer is a fingerprint — a small file a human writes by hand, declaring what a correct restore must contain. Not generated. Not exported from the database. Declared, the way you would declare what matters before a trip: passport, keys, the one document you cannot replace. His fingerprint has four components:

• An expected file list. The files (or tables, or object prefixes) that must be present in the restore. Not all of them — the ones whose absence would mean the backup missed its point.
• One file's sha256. Pick one file whose content you can predict — a config file, a seed data file, a known document — and record its hash. The restore must reproduce it bit for bit.
• A row count. One table that matters, with a count (or a sane range) a human states from knowing the business, not from querying the backup. Orders should be in the hundreds of thousands by now; if the restore says 41, something is deeply wrong even if the restore "worked."
• One config value. A single setting whose correctness you can declare: the application's environment name, a feature flag that must be on, the schema version the code expects. Config drift is how "successful" restores fail at boot time.

Four declarations. Small enough to write in fifteen minutes, small enough to read at a glance, and — this is the whole design — none of them can be produced by the system they verify. The fingerprint stands outside the backup pipeline and judges it.

An example, as a plain file kept with the runbook:

# restore-fingerprint.yaml — declared by hand, re-declared quarterly
system: billing-db
expected_files:
  - /srv/app/config/app.env
  - /srv/app/data/seed-countries.csv
sha256:
  /srv/app/config/app.env: "9f2c…e41a"   # recorded at declaration time
row_counts:
  orders: ">= 400000"                      # stated from the business, not the dump
config_values:
  schema_version: "2026.09"                # what the current code expects
declared: 2026-11-02
next_review: 2027-02-02

The exact format does not matter. What matters is the direction of the arrow: expectations flow from a human's knowledge *into* the check, never from the checked system *outward*.

Why insist a human writes it? Because the declaration is not data entry — it is a review disguised as data entry. To write the fingerprint, someone has to sit down and answer questions that no monitoring dashboard asks: *What in this system would we actually miss? Which table is the business? Which config value, if wrong, turns a good restore into a bad morning?* Those answers live in people's heads and in the incident history, not in the database.

In dafili's practice, the human re-declares the fingerprint quarterly — deliberately on the same cadence as the full restore drill. Systems drift legitimately over a quarter: new tables appear, row counts grow, config values change on purpose. The quarterly re-declaration is the moment a human looks at that drift on purpose, in calm conditions, and says: *yes, this is what correct looks like now.* Skip it, and the fingerprint slowly becomes a declaration about a system that no longer exists — which is its own quiet failure mode, and exactly why the declaration cannot be a one-time setup task.

This is also why automating the declaration defeats the purpose. A generator that rebuilds the fingerprint from production every night produces a fingerprint that is always exactly right and therefore says nothing: it re-declares whatever the system currently is, including its mistakes. The value of the fingerprint is precisely that it can *disagree* with the system — and that a human chose the terms of the disagreement in advance.

Everything after the declaration should be boring, mechanical, and nightly. Restore the latest backup into a scratch environment, extract the actuals, and diff them against the fingerprint:

#!/usr/bin/env bash
# compare-restore.sh — runs after the nightly restore-to-scratch
set -euo pipefail

FAIL=0

# 1. Expected files present?
while read -r f; do
  [ -f "/tmp/restore-test$f" ] || { echo "MISSING: $f"; FAIL=1; }
done < <(yq -r '.expected_files[]' restore-fingerprint.yaml)

# 2. The declared hash matches?
sha256sum -c <<<"$(yq -r '.sha256."/srv/app/config/app.env"' restore-fingerprint.yaml)  /tmp/restore-test/srv/app/config/app.env" \
  || { echo "HASH DRIFT: app.env"; FAIL=1; }

# 3. The declared row count is in range?
ORDERS=$(psql restore_test -tAc "SELECT count(*) FROM orders;")
[ "$ORDERS" -ge 400000 ] || { echo "ROW COUNT DRIFT: orders=$ORDERS"; FAIL=1; }

# 4. The declared config value survived?
psql restore_test -tAc "SELECT value FROM settings WHERE key='schema_version';" \
  | grep -qx "2026.09" || { echo "CONFIG DRIFT: schema_version"; FAIL=1; }

exit $FAIL

The details will differ with your stack — the pattern is the point. Automation compares; it does not get an opinion. When the comparison fails, it shouts: a failed fingerprint comparison is a page, not a log line, because it means one of two things, both important — the backup is wrong, or the world changed and nobody re-declared it.

That second case deserves respect rather than dread. Most fingerprint failures, in a healthy team, will be legitimate change: a migration added a table, the row count outgrew the declared range, a config value rolled forward. The correct response is not to silence the check but to re-declare deliberately — a human reviews the drift, confirms it is intended, and updates the fingerprint as its own small, visible act.

Contrast that with the alternative everyone actually lives with: expectations embedded in the check script, edited quietly at 2 a.m. to make the red go away, until the check asserts nothing. The fingerprint does not prevent someone from weakening a check. It makes the weakening a declared event, in a file with a date on it, that a second person can review. That is the difference between a system that learns and a system that merely quiets down.

There is one more layer, and it might be the most practical idea in the whole thread. Automated nightly comparisons are excellent — right up until the human parts of the practice quietly die. The quarterly manual restore is the first thing skipped in a busy quarter. Then it is skipped twice. Nothing alarms, because the automation is still green; the drill's *judgment* — a person restoring by hand, noticing the weird thing no fingerprint anticipated — has been absent for a year and nobody knows.

Dafili's countermeasure is to monitor the drill itself with a gauge: days_since_last_successful_manual_restore, and to page when it crosses about 120 days. Not a ticket. A page — the same severity as a failed backup, because that is what it is: past roughly one quarter, you no longer have evidence that a human can restore this system, only evidence that a script once could.

In Prometheus terms it is a timestamp gauge the drill updates when — and only when — a manual restore passes its fingerprint comparison:

# Written by the drill checklist's final step, after a human-run
# restore passes the fingerprint comparison:
echo "last_successful_manual_restore_timestamp $(date +%s)" \
  > /var/lib/node_exporter/textfile/restore_drill.prom

# Page when no successful manual restore in ~120 days
(time() - last_successful_manual_restore_timestamp) / 86400 > 120

Readers of the ladder post will recognise the shape: it is the dead-man's-switch from that article, pointed at the humans. There, the monitored absence was *verification success* — page when the checks stop succeeding. Here, the monitored absence is the *drill itself* — page when the people stop practising. Automation watches the backups; the gauge watches the watchers' habits. A restore you have not performed in four months is a hypothesis, and the gauge exists to say so out loud while it is still cheap to fix.

One honest open question, because it came up in the thread and I do not have dafili's answer yet: where should the fingerprint be stored? I asked him directly in the conversation; while his reply is pending, the trade-offs are worth stating plainly, because the obvious answer has a thorn in it.

• With the backup configuration is convenient — the drill finds it without ceremony. But it risks the fingerprint drifting *with* the thing it verifies: the same change that breaks the backup can quietly rewrite the expectations, the way a scale that recalibrates itself to whatever it weighs can never report a wrong weight.
• In a separate repository, owned like a runbook, keeps the declaration independent and reviewable — changes arrive as commits a second person can see. The cost is one more place to keep current, and a drill that must fetch its expectations from somewhere it trusts.
• On paper, or in the incident binder, sounds like a joke until you remember what the fingerprint is for: it is the declaration that survives the system. Four values fit on an index card.

Wherever it lives, the property to protect is independence: the fingerprint must not be writable by the pipeline it judges. When dafili answers, this section gets an update — his reasoning has improved this series twice already.

• A fingerprint file exists for each system that matters, written by hand — not generated from the system it verifies
• It declares all four: an expected file list, one file's sha256, one row count (or range), one config value
• A human re-declares it quarterly, on the same cadence as the full restore drill
• The nightly drill restores to scratch and compares actuals against the fingerprint automatically
• A failed comparison pages — and legitimate drift is resolved by deliberate re-declaration, never by editing the check to be quiet
• A days_since_last_successful_manual_restore gauge (or equivalent) exists, with a page at about 120 days
• The fingerprint is stored somewhere the backup pipeline itself cannot rewrite
• Expectations in older checks that were derived from the checked system have been replaced with declared ones

Isn't this just the level-4 verification from the previous post? It is level 4 done honestly. The ladder says *restore and verify*; the fingerprint answers the question that post left open — verify *against what?* If the answer is "against values taken from the restore itself," the verification is circular. Declared expectations are what break the circle.

Four checks seem too few. Shouldn't the fingerprint cover everything? Coverage is the automation's job — checksums, listings, and sampled reads already sweep the whole artifact nightly. The fingerprint has a different job: to be an independent witness. A witness who testifies about everything testifies about nothing; four declarations a human can hold in their head, and re-declare with full attention each quarter, beat four hundred generated ones nobody reads.

What if the business can't state a row count in advance? Declare a range or a floor, stated from business knowledge ("orders grow by roughly ten thousand a month; a restore below last quarter's count is wrong"). Precision is not the point — independence is. An expectation that is roughly right and genuinely external catches the catastrophic wrongness the drill exists to catch.

Does the gauge replace scheduling the quarterly drill? No — it catches the schedule failing. Calendars get ignored under load; that is precisely the failure the gauge exists to surface. Schedule the drill *and* monitor its successful completion, the same way you schedule backups *and* monitor verification success.

The ladder from the previous post — exists, intact, readable, restored — tells you how to check a backup. Dafili's fingerprint tells you what makes the checking mean something: expectations a human declared, standing outside the system, that the system cannot rewrite to flatter itself. *You cannot automate the declaration, only the comparison.* So declare by hand, re-declare quarterly, compare nightly, shout on drift — and keep a gauge on the drill itself, paging at about 120 days, so the practice cannot die in silence.

Credit where it is due: the fingerprint doctrine, the declaration/comparison line, and the drill gauge all come from dafili, in the comment thread under "My Backup Exited 0 for Three Weeks and Nothing Paged Me" — the same thread that produced the ladder post and corrected it. His follow-up, "Three checks that look like they passed but validated nothing", is the natural next read. This series keeps getting better because a reader keeps arguing with it in public. Long may that continue.