When something goes wrong — recovery

What can actually break

The point isn't that all of these will happen. The point is that none of them feel survivable when they happen and you don't have a plan. The procedures below are the plans.

The six-leg push, applied to ~/.claude/

WholeTech already runs a 6-leg backup pattern for the websites tree (droplet, B2, GitHub, Drive, HS NAS, CC NAS). Apply the same pattern to your Claude config and you get disaster recovery basically for free.

The hierarchy is:

1. Last 24 hours → secondary PCs (fastest, online).
2. Last week → GitHub repo history (versioned, diffable).
3. Last month → B2 daily snapshots.
4. Older → Drive + NAS cold storage.

Laptop is gone

Treat as compromised. The threat isn't "they might find the files" — full-disk encryption handles that. The threat is "they might be holding a logged-in machine right now." Lock down accounts the laptop had access to, in order of value.

1. Revoke the Anthropic session. claude.ai → Settings → Sessions → revoke the missing device. Forces re-login.
2. Rotate the SSH key. On the droplet: edit ~/.ssh/authorized_keys, delete the laptop's public key. From any other machine in the fleet that still has access. Verify: ssh -i missing-key root@wholetech.com should now fail.
3. Rotate the Anthropic API key if one was on the laptop. console.anthropic.com → API Keys → revoke the named key, issue a new one. Update on every other machine that needed it.
4. Rotate the GoDaddy API key if godaddy.env was on the laptop. developer.godaddy.com → revoke + reissue. Update ~/.secrets/godaddy.env on the remaining PCs.
5. Rotate the GitHub PAT if one was on the laptop. github.com/settings/tokens → revoke. Reissue, redeploy.
6. Rotate the rclone remotes if rclone.conf was on the laptop. Each remote (B2, Google, etc.) has its own console; rotate the keys there.
7. Find My (Apple) or Find My Device (Google) — lock and erase remotely if it's a Mac or Chromebook.
8. Replace the laptop. From a fresh machine: re-run the bootstrap procedure (clone the dotfiles repo, run claude login, replace the per-PC secrets from your secrets vault).

Per-machine SSH keys make this much easier

If every machine has its own SSH key (laptop has laptop-ed25519, desk PC has desk-ed25519, etc.), rotation is one line in ~/.ssh/authorized_keys on the droplet. If they all share one key, you're rotating one key on every machine. Always per-machine.

Anthropic account is locked

Rare. Happens if Anthropic flags unusual activity, payment failure, or a violation. The frustrating part: there's no instant fix; you're on Anthropic's support timeline. But you can keep working in the meantime.

1. Contact Anthropic support immediately. support.anthropic.com — describe what you were doing right before the lock. Honest, factual.
2. Don't try to log in repeatedly — that often extends the lock.
3. Switch local Claudes to a different account temporarily. If you have a team/business workspace, use that. If not, a colleague or family account can keep you running for non-sensitive work.
4. Pay-per-use via the Anthropic API. A different Anthropic API account (or your team account's API key) keeps Claude Code working — codex login --api-key or ANTHROPIC_API_KEY=.... The CLI keeps working even if your ChatGPT-style web account is locked.
5. Mobile and web are dead. Until the lock lifts. Plan around that.
6. Document for review. What you were doing, when, on which IP, with which key. Send the bundle to support; speeds up resolution.

~/.claude/ is broken

A bad JSON edit; a Dropbox conflict; a permissions glitch; an OS upgrade that scrambled file ownership. The most common shape: Claude starts and immediately errors on parsing settings.json, or hooks silently stop firing.

Diagnosis first

1. Try claude --version. If this works, the CLI binary is fine. If not, reinstall.
2. Try cat ~/.claude/settings.json | jq .. If jq errors, your settings.json is invalid JSON. That's by far the most common cause.
3. Try claude --debug. The verbose logs usually point at the broken file.
4. Look at the auto-backup directory. Many Claude Code versions write ~/.claude/backups/settings.json.<timestamp> before saving. Roll back to the most recent.

If you have the git repo

$ cd ~/.claude
$ git status                      # see what changed
$ git checkout HEAD settings.json   # revert the file
$ git log -p settings.json         # see history; pick a commit to roll to

If you don't

# on another machine in the fleet, copy the file across
$ scp other-pc:.claude/settings.json ~/.claude/settings.json

# or pull from your B2 snapshot
$ rclone copy b2:walhus-backups/claude-config/latest/settings.json ~/.claude/

Nuclear option

Move the whole ~/.claude/ aside (don't delete — you might need to copy memory back later), re-run claude login to create a fresh one, then layer your dotfiles back on top.

$ mv ~/.claude ~/.claude.bak
$ claude login                       # creates a fresh ~/.claude/
$ rm -rf ~/.claude
$ git clone <repo> ~/.claude
$ cp ~/.claude.bak/auth.json ~/.claude/   # or just re-login

Lost or leaked API keys

1. Revoke the key. console.anthropic.com → API Keys → revoke the named key. Immediately. Don't wait to figure out if it actually leaked.
2. Issue a replacement, with the same scope. Name it so you'll know what it is in six months: laptop-2026-05, droplet-cron, ci-builds.
3. Update everywhere it was used. Each machine's ANTHROPIC_API_KEY env var, each CI job's secret, the droplet's /etc/environment, any rclone-driven config sync.
4. Audit recent usage. The Anthropic console shows usage by key. Look for activity that wasn't yours; that tells you whether the key actually was used by an attacker.
5. Set a spend cap on the new key if you didn't have one. Limits the blast radius of the next leak.

An MCP server stopped working

Almost always auth-related. Google OAuth tokens refresh quietly until they don't; GitHub PATs expire; database passwords change. Symptoms: Claude says it can't reach that MCP tool, or the tool list is shorter than usual.

1. Identify which server. Run /mcp inside Claude — the status of each is shown. Red/yellow = auth issue.
2. Re-authenticate. For OAuth-based ones (Google, Slack), claude mcp auth <server-name> opens the auth flow. For token-based ones (GitHub, Postgres), update the env var or the secret file.
3. Restart the Claude session. The MCP server is a subprocess; it reads the env at startup. Quit and reopen.
4. Check ~/.claude/mcp-needs-auth-cache.json — Claude tracks which servers are pending auth here. If it's stale, delete it; it regenerates.

The renewal cadence to remember

The droplet is down

DigitalOcean (or whoever) is having a bad day; the droplet was rebuilt accidentally; the disk filled and nginx OOM'd. Distinguish "down" (will come back) from "lost" (need to rebuild).

If "down"

1. Check provider status page. If known incident, wait.
2. If droplet is up but services are down, SSH in. Look at journalctl -u nginx -n 100.
3. Most common: disk full. df -h → journalctl --vacuum-size=200M to free /var/log; trim old /var/www/*/sessions/ dirs.

If "lost" — rebuild from scratch

1. Spin up a fresh droplet. Same size, same OS, same hostname (for DNS continuity).
2. Update DNS A records if the new IP differs. GoDaddy API per the runbook.
3. Restore the websites tree. From your most recent backup leg (CC NAS or B2). rclone copy b2:walhus-backups/var-www/ /var/www/.
4. Reinstall nginx and certbot. apt install nginx certbot python3-certbot-nginx. Restore vhosts from the backup of /etc/nginx/sites-available/.
5. Re-issue certs. certbot --nginx -d <domain> for each — DNS already points at the new IP, so the HTTP-01 challenge works.
6. Reinstall Claude Code. npm i -g @anthropic-ai/claude-code. Set ANTHROPIC_API_KEY in /etc/environment.
7. Restore the cron jobs. From the backup of /etc/cron.d/ or the user crontab dump.
8. Verify. curl -sI https://<each-site> → all 200. ssh root@wholetech.com 'claude --version'.

The NAS is dead

Drive failure; controller failure; power surge. Less acute than a dead droplet (no public services lose access) but slower to fully recover (drives to source, RAID to rebuild, hours to days of resync).

1. Identify scope. Single drive failure → RAID handles it; rebuild after replacement. Multiple drives or controller → restore from other legs.
2. Pause writes. Stop scheduled jobs that mirror to this NAS. They'll queue locally; resume when the NAS is back.
3. Restore from B2. Your overnight cloud backup is the next-best source. rclone copy b2:walhus-backups/ /mnt/nas/restore/.
4. Restore from the other NAS. If HS NAS is dead, CC NAS still has the mirror (and vice versa). Two-NAS topology is the case where this kind of failure stays survivable.
5. Replace, rebuild, resume. Source drives, rebuild the array, re-establish the mirror jobs, verify checksums.

The Claude angle: if your task queue lives on this NAS, queue work pauses until restore. Workers should keep heartbeating; new tasks queue at the orchestrator until the queue layer is back.

Restoring an auto-memory file

A particularly insidious failure mode: a memory file gets overwritten or deleted, and you don't notice for a week — by which time Claude has been operating without that context. By then the symptom is "Claude keeps making the mistake I corrected last month."

If memory is in git (recommended)

1. cd ~/.claude
2. git log -p projects/<slug>/memory/ — see what changed, when.
3. git checkout <good-commit> -- projects/<slug>/memory/<file>.md
4. Verify MEMORY.md still references the restored file. If you'd already removed the index entry, restore that too.

If memory is in OneDrive / Dropbox

Both services keep file version history. OneDrive: right-click the file in Explorer → Version history. Dropbox: right-click → Version history (or web UI → the file → Version history). Restore the version from before the mistake.

If memory is gone everywhere

This is why I keep saying "version your memory." If you've lost a memory and have no version history, reconstruct from:

• Your conversation history: scroll back in claude.ai or in CLI ~/.claude/projects/<slug>/history.jsonl — find the session where the memory was first written; Claude probably summarised it back to you at the time.
• The PR or commit where it was first introduced: often memories about a codebase have a corresponding commit you can read.
• Your physical notes, if you have any.

Logging for early warning

The cheapest disaster is the one you catch a minute after it happens, not a week. Three lightweight monitors that pay back hundreds of times their setup cost:

1. Heartbeat per machine

Every machine writes $(hostname) $(date) to a shared file (NAS or droplet) every 10 minutes. A separate process checks the file every hour; any host whose heartbeat is >30 min old fires an alert. Catches "the scheduled task on the laptop quit working" within an hour.

# cron entry, every 10 minutes
*/10 * * * * echo "$(hostname)  $(date -Iseconds)  alive" >> /mnt/nas/fleet-heartbeats.log

2. settings.json validity check

Once a day, on each machine, validate that ~/.claude/settings.json parses. If it doesn't, ping the notification channel.

*/60 * * * * jq . ~/.claude/settings.json > /dev/null 2>&1 || curl -d "settings.json broken on $(hostname)" ntfy.sh/walhus-claude

3. Disk space

The droplet running out of disk has been the cause of more weekend incidents than anything else. df -h at 09:00 daily; warn if /var is >80%.

The fire drill

Once a quarter, practise. Pick a scenario, set a timer, run the procedure. You'll find out which steps are stale, which secrets you can't actually locate, which backups didn't actually back up.

Three drills worth running

What goes wrong in recovery

The rest of the hub