2 · Roll back

TL;DR

Objective — when you can’t fix production fast enough, restore the last good release instead. The deployer keeps previous releases on disk, so rolling back is usually a 30-second symlink switch — no code changes, no merge.

User part + done-when

👤 User part — the production rollback/deploy itself is human-gated: a person runs the dep rollback / dep deploy production command against the live server. The user also confirms in a browser that the restored release works, and decides whether the breakage involves a database migration (which changes the whole approach). The agent prepares the git side (listing tags, creating + pushing a rescue branch) but does not switch the live release — same gating as branch-protection and live-payment actions.

Done when production serves a known-good release, monitoring is clean, and the incident is documented so the real fix can follow via the normal flow.  ·  ⏱ ~5–15 min  ·  🔀 mixed (🤖 git prep, 👤 production deploy + verify + DB call).

Background

Rolling back feels like giving up, but it’s the responsible move when a proper fix would take too long. A deploy doesn’t overwrite the old code — the deployer publishes each release into its own folder and just points a current symlink at the newest one. “Rollback” means pointing that symlink back at the previous folder. The old code is still sitting there, intact.

That’s why rollback is faster and safer than a hotfix: there’s no new code to get wrong. The one thing it does not undo is database migrations — if a migration changed the schema, switching code back won’t reverse it, and forcing it can lose data. That caveat drives the decision flow below.

Two more things a code rollback leaves untouched (by design): the deployer keeps storage/ as a shared directory symlinked across releases, so user uploads and the storage/installed install marker survive the rollback — switching the current symlink does not touch them. That’s what you want (you don’t lose uploads when you roll back), but it means rollback is a code time-machine only, not a data one.

Read the production row in Zaj-PROJECT.md before preparing a rollback. It records the production deploy target, SSH alias, deploy path, PHP binary, DB name, backup pointer, and any non-production branches that need the eventual fix. If the rollback uncovers a changed deploy path, runtime version, SSH alias, or DB fact, update Zaj-PROJECT.md after the live site is stable.

Before calling SSH credentials missing, discover existing aliases:

grep -nE '^Host |HostName|User|Port|IdentityFile' ~/.ssh/config 2>/dev/null
SSH_PRODUCTION_ALIAS="<production-alias-from-Zaj-PROJECT>"
PHPBIN="<production-cli-php-from-Zaj-PROJECT>"
ssh -o BatchMode=yes -o ConnectTimeout=8 "$SSH_PRODUCTION_ALIAS" "pwd && $PHPBIN -v | head -1"
# Expected: exit 0; alias/host/user/port match the production row

flowchart TD
    A{Production broken} --> B{Was it the<br/>LAST deploy?}
    B -->|Yes| R[dep rollback production<br/>30 sec]
    B -->|No| T{Know the last<br/>good tag?}
    T -->|Yes| D[dep deploy --tag=vX.X.X]
    T -->|No| RB[Rescue branch<br/>from a known-good tag]
    A --> M{Is it a DATABASE<br/>migration issue?}
    M -->|Yes| W[Do NOT just roll back code —<br/>see the migration caveat]

Steps

1. Roll back the code

Pick the lightest option that fits. Option 1 covers most incidents — reach for 2 or 3 only when the bad release isn’t the most recent one.

Who does this

👤 User — each option deploys to the live production server, so a person runs the dep rollback / dep deploy production command itself, then verifies the site works. The agent prepares (reads logs, finds the good tag, creates a rescue branch and pushes it) but does not switch the live release — production deploys are human-gated, the same way branch-protection and live-payment actions are fenced from the agent.

1. Option 1 — deployer rollback (fastest). 👤 USER step — production is human-gated. Use when the last deploy caused the issue. Switches the symlink to the previous release — instant, no code changes. Who runs this — 👤 you (production is human-gated): a person runs the rollback against the live server.

   dep rollback production
   # Expected: the live symlink now points at the previous release; site recovers

   • ✅ 👤 You ran the rollback; production loads a working release within ~30 seconds and you confirm it in a browser.

2. Option 2 — deploy a specific tag. 👤 USER step — production is human-gated. Use when you know which version was last stable but it isn’t the immediately previous release. The agent can list the tags; 👤 you run the production deploy.

   git tag --list "v*" --sort=-v:refname | head -10
   dep deploy production --tag=v[LAST_GOOD]
   # Expected: production now runs the named known-good tag

   • ✅ The chosen tag is live (deployed by 👤 you) and you confirm the site works.

3. Option 3 — rescue branch (full rollback). 👤 USER step — production is human-gated. Use when you need to go back several versions or the options above don’t work. The agent can create + push the rescue branch and tag; 👤 you run the dep deploy production against the live server.

   git checkout -b rescue-from-v[LAST_GOOD] v[LAST_GOOD]
   git push origin rescue-from-v[LAST_GOOD]
   dep deploy production --branch=rescue-from-v[LAST_GOOD]
   git tag rescue-$(date +%Y-%m-%d) && git push origin rescue-$(date +%Y-%m-%d)
   # Expected: a rescue branch from the known-good tag is deployed and tagged

   • ✅ The rescue release is live (deployed by 👤 you), tagged, and you confirm the site works.

The decision tree is simple: last deploy broke it → Option 1; you know the good tag → Option 2; you need to go back far → Option 3.

2. Handle the database caveat

A code rollback does not undo migrations. If a migration is what broke production, switching code back will not fix it — and the wrong move here causes data loss.

Who does this

👤 User decides and runs any database recovery. Migrations that added a table or column can’t be auto-reversed without judgment — a person must confirm it’s safe or restore from backup.

1. Back up the database FIRST — rollback net before any reversal. migrate:rollback is a destructive op (a down() that dropped a column/table cannot be un-dropped). Take a verified dump you can restore from before touching the schema.

   # Herd Pro ships MariaDB → the dump tool is mariadb-dump (mysqldump was renamed; alias it if you prefer).
   # On the server, run it against the production DB (or via the hosting panel's export).
   mariadb-dump -u [DB_USER] -p [DB_NAME] > "backup-$(date +%Y%m%d-%H%M%S).sql"   # or: mysqldump ...
   ls -lh backup-*.sql   # Expected: a non-empty .sql dump exists before any migrate:rollback

   • ✅ A verified, non-empty backup exists before any migration is reversed. If the dump tool is missing, install it (Prerequisites) — never skip the backup because the tool wasn’t ready.

2. Check what would roll back, then choose a safe recovery. Only roll back a migration batch if you’re certain it drops nothing you need.

   php artisan migrate:status
   php artisan migrate:rollback --pretend   # the SQL the reversal WOULD run — no DB change; read it first
   # Option A: rollback last batch — ONLY after the backup above AND --pretend confirms it loses no data
   php artisan migrate:rollback
   # Option B: restore from the backup (the dump above, hosting panel, or your backup system)
   # Expected: migrate:status lists which migrations are in the last batch

   • ✅ Either the unsafe batch is safely reversed (backup taken first), or the database is restored from a known-good backup.

Migration rollback can delete data — back up first, and never migrate:fresh on live data

If a migration added a column or table, rolling it back removes that column or table — and everything in it. Take the verified dump in step 1 first; when in doubt, restore from backup instead of running migrate:rollback. Never run php artisan migrate:fresh / migrate:reset / wipe against staging or production — they drop all tables and are local-rebuild-only commands.

3. Document and plan the real fix

A rollback is temporary — it stops the bleeding but leaves the root cause unfixed. Capture what happened while it’s fresh, then fix it properly through the normal staged flow.

1. Write a short incident note and queue the real fix. Record what broke, why, and how you restored it, then plan the permanent fix via the normal flow.

   • ✅ An incident note exists (what happened · root cause · resolution · action items), and the proper fix is queued for the staged develop → selected non-production target → production path, not another rushed hotfix.

The rollback bought you time; it didn’t buy you a fix. Return to develop, find the real root cause, and ship it the safe way via Continuous Deploy.

Checklist

Do not mark this step done until every box below is checked.

• 🔀 Code restored — 🤖 agent prepared the git side (tags / rescue branch); 👤 you ran the dep rollback / dep deploy production (production is human-gated) so a known-good release is live.
• 👤 Verified live — production loads, Sentry is clean, users can access the app and pay.
• 🔀 Database handled — if a migration was involved, a verified DB dump was taken first, then it was safely reversed (migrate:rollback --pretend reviewed) or restored from backup (👤 decision). No migrate:fresh/reset/wipe against live data.
• 👤 Incident documented — what happened, root cause, and resolution are written down.
• 🤖 Real fix queued — the permanent fix is planned for the normal staged flow, not another rushed hotfix.

That’s the whole hotfix workflow

You’ve covered both moves — fix forward and roll back. Head back to the Emergency Hotfix overview for the decision flow, or to Laravel/CodeCanyon for the other four workflows.