YourBackupIsLyingtoYou

I opened my terminal on a Tuesday morning expecting six days of clean offsite git commits. Every local log was green. The launchd agent was firing on the dot every 15 minutes. The cron job recorded a clean exit code zero across 576 consecutive runs.

Not a single byte had left my machine.

The commits had succeeded locally. The remote push had died silently on day one, choked on a 112MB screen capture someone dragged into an asset subfolder. GitHub’s pre-receive hook had slammed the door on the oversized blob, the script had swallowed the stderr output in a background subshell, and the automation spent the rest of the week cheerfully reporting that it had done its job.

Activity is not outcome.

# What the dashboard checked:
pgrep -f "vault-sync" && echo "ONLINE"

# What actually mattered:
git rev-list --count origin/main..main

The Illusion of Green Checks

Personal infrastructure suffers from an observability disease that production systems cured years ago: measuring motion instead of state. When an enterprise database backup fails, PagerDuty wakes up three people. When a personal workstation sync fails, it fails in the dark.

The sync script was checking whether git commit succeeded. Because there were local edits, git commit -m "vault snapshot" created a commit object without friction. The script then invoked git push origin main in the background with standard output silenced.

To the process table, the job ran. To the logfile, the job finished. To the user, everything was protected. In reality, six days of writing and architecture work existed on exactly one spinning NVMe drive sitting on a desk.

History Doesn’t Care About Your Tip

The immediate instinct when GitHub rejects a 100MB blob is to fix the tip: remove the file from the workspace, add it to .gitignore, and run another commit.

rm assets/recordings/demo-screen-capture.mov
echo "assets/recordings/*.mov" >> .gitignore
git commit -am "untrack oversized demo video"
git push origin main

It fails again instantly.

A git push transfers commits, not a working directory diff. Untracking the file at the latest commit does not remove the compressed object from the six commits sitting right behind it in the unpushed pipeline. The blob is baked into the immutable SHA DAG. Every subsequent run re-attempts to push that same blocked chain, and every run eats the same rejection.

To unstick the sync without blowing away history, you have to rip the object out of the unpushed commits entirely or rebase:

git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch assets/recordings/demo-screen-capture.mov' \
  --prune-empty --tag-name-filter cat -- origin/main..main

The Three Rules of Dumb Automation

Three operational principles came out of that six-day blind spot:

1. Test the Invariant, Never the Process

Don’t check if the script executed. Check if the divergence between local and remote is zero. The health check is one line:

UNPUSHED=$(git rev-list --count origin/main..main)
if [ "$UNPUSHED" -gt 0 ]; then
    notify-push-lag "$UNPUSHED commits pending"
fi

If that counter ticks above zero for more than two cycles, the system is down regardless of what the logs claim.

2. Alert on Outliers, Throttle the Panic

A naive alert script creates a second failure mode: the false alarm. Close your laptop lid, put it in a backpack, or hop on an airplane, and a dumb timer starts blasting push-failure notifications because Wi-Fi is unreachable.

Alert design requires two gates: verify network reachability first, and throttle alerts so you receive exactly one high-priority ping when an actual wall is hit—not a notification barrage every quarter-hour.

3. Fail Loud or Don’t Automate

Silent fallbacks are technical debt that collects interest in disaster scenarios. If a script cannot guarantee delivery, its duty is to scream.

Honest Caveats

• Network-Disconnected Laptops: If you’re on a flight or working off-grid on a ranch, your push divergence will naturally climb. A naive script that pages you every fifteen minutes when offline will train you to ignore your own alarms. The invariant check must verify external internet ping before tripping alerts.
• Git Is Not a Block-Level Backup: Using git for file sync protects text and code, but it is not a disaster-recovery strategy for binary assets, OS state, or database partitions. Keep your git vaults separate from proper restic or Time Machine backups.

Done is better than perfect, but a backup that doesn’t push is not done. It’s just a local copy with extra steps.