Skip to content
prod 352bb92
Browse

1 · Hotfix forward

Objective — patch a live, broken app the fast-but-safe way: branch from the production state (not develop), make the smallest possible fix, deploy straight to production skipping staging, then back-merge so the fix isn’t lost on the next release.

The thing that makes a hotfix different from a normal deploy is where you branch from. A normal change starts on develop and flows through the selected non-production target from Zaj-PROJECT.md before production. A hotfix can’t wait for that — production is already broken, and develop may contain half-finished work you don’t want live. So you branch straight from production, fix only the one thing, and deploy directly.

That speed has a cost: the fix now lives only on the production line, not in develop. If you stop there, the next normal release silently overwrites your fix and the bug comes back. That’s why Phase 3 (back-merge) is not optional — it’s the step that makes the fix permanent.

Read Zaj-PROJECT.md before touching production. It tells you the production SSH alias, deploy path, PHP binary, branch model, and any extra non-production branches (staging, qa, uat, client-demo) that need the fix after production is stable. If the hotfix changes runtime/dependency/deploy facts, rerun the relevant parity check and update Zaj-PROJECT.md plus .tool-versions; otherwise mark parity N/A with evidence in Zaj-PROGRESS.md.

Before calling SSH credentials missing, discover existing aliases:

Terminal window
grep -nE '^Host |HostName|User|Port|IdentityFile' ~/.ssh/config 2>/dev/null
SSH_PRODUCTION_ALIAS="<production-alias-from-Zaj-PROJECT>"
PHPBIN="<production-cli-php-from-Zaj-PROJECT>"
ssh -o BatchMode=yes -o ConnectTimeout=8 "$SSH_PRODUCTION_ALIAS" "pwd && $PHPBIN -v | head -1"
# Expected: exit 0; alias/host/user/port match the production row
flowchart LR
P[production] -->|branch from| H[Hotfix/fix-description]
H -->|deploy direct| P2[production live]
H -->|back-merge| D[develop]
D --> S[non-production branches]
P2 -->|merge| M[main]

Spend two minutes confirming it’s a true emergency and that you understand the bug. A wrong fix deployed in a panic is worse than the original break.

  1. Read the production error log and recent deploys. Find the actual error before writing any fix.

    Terminal window
    ssh [SSH_PRODUCTION_ALIAS] "tail -50 [DEPLOY_PATH]/storage/logs/laravel.log"
    git log --oneline production -5
    # Expected: a stack trace / error message, and the last 5 production commits
    • ✅ You can name what’s broken and, if a deploy caused it, which one.
  2. Decide: hotfix or roll back. If you can fix it in under an hour, continue here. If not — or if the last deploy is the culprit — switch to Roll back and restore the last good release first.

    • ✅ You’ve chosen to fix forward, and you know the fix.

Branch from production itself — not develop — so your fix sits on exactly the code that’s live, with nothing extra coming along for the ride.

  1. Branch from the live production state.

    Terminal window
    git checkout production && git pull origin production
    git checkout -b Hotfix/fix-description
    # Expected: now on Hotfix/fix-description, branched off current production
    • git branch --show-current prints Hotfix/fix-description.
  2. Make the minimal fix, test locally, and commit. Change only what’s needed to stop the bleeding — no refactors, no extras.

    Terminal window
    # find and fix the bug, then a quick local test
    git commit -am "🔧 🟦 T2 Fix-Bug: Fix [description]"
    git push origin Hotfix/fix-description
    # Expected: one small commit pushed to the hotfix branch
    • ✅ The fix is a small, scoped diff and is pushed to the remote.

A hotfix that touches twenty files isn’t a hotfix — it’s a feature in disguise. Keep the diff small enough to review at a glance under pressure.

This is the one workflow where you skip staging on purpose. Production is already broken, so the normal staged path would only extend the outage. Merge the hotfix into production and deploy.

  1. Merge and deploy to production. 👤 USER step — production is human-gated. If the fix includes a migration, confirm the verified DB dump from §2 exists before deploying — the deploy runs migrations. Who runs this — 👤 you (production is human-gated). The agent prepares the hotfix branch, the merge, and the DB/vendor guards; a human runs the actual production push + deploy. Even under incident pressure, agents stay fenced from production deploys (real-user / real-money actions) — the same gating as branch-protection and live payments.

    Terminal window
    git checkout production
    git merge Hotfix/fix-description
    git push origin production
    dep deploy production
    # Expected: deployer publishes a new release, runs migrations, and switches the live symlink.
    # storage/ is a shared dir → uploads + the storage/installed marker carry over untouched.
    • ✅ 👤 You ran the production deploy; it completes without errors and the new release is live (DB dump on hand if a migration ran).

Now hand off to a person for the one check an agent can’t truly make — does it actually work for a real user?

Open production in a browser, reproduce the original failing action, and confirm it now succeeds before moving on.

The fix is live but fragile — it exists only on the production line. Tag it, then fold it back into every branch so the next normal release can’t undo it.

  1. Tag the fix and propagate it to main, develop, and every non-production branch listed in Zaj-PROJECT.md.

    Terminal window
    NON_PROD_BRANCH="staging" # example; repeat for each selected non-production branch in Zaj-PROJECT.md
    git tag v[VERSION]-[next letter] # e.g. v1.0.0-a → v1.0.0-b
    git push origin v[VERSION]-[next letter]
    git checkout main && git merge production && git push origin main
    git checkout develop && git merge Hotfix/fix-description && git push origin develop
    git checkout "$NON_PROD_BRANCH" && git merge develop && git push origin "$NON_PROD_BRANCH"
    # Expected: the fix now exists on main, develop, and every configured non-production branch
    • git branch --contains Hotfix/fix-description lists develop (and the fix is on main plus every configured non-production branch).

This back-merge is the difference between fixing the bug once and fixing it forever. Skip it and the bug reappears on the next deploy from develop.

Once a person has confirmed everything is merged and stable, clean up:

Terminal window
git checkout develop
# then delete the hotfix branch (local + remote) once 👤 confirms it's safe

Do not mark this step done until every box below is checked.

  • 🤖 Assessed — production error read; confirmed it’s a true emergency and the fix is understood.
  • 🤖 Branched from production — on Hotfix/fix-description, off the live state (not develop).
  • 🔀 Minimal fix shipped — 🤖 agent prepared the small scoped diff and merge; 👤 you ran the production push + dep deploy production (production is human-gated) and it succeeded.
  • 🔀 DB / vendor guards cleared — if the fix added a migration, a verified DB dump was taken first and migrate --pretend/Atlas lint reviewed; if it touched vendor/, the package was clean-reinstalled (not composer install) and re-scanned for backdoor signatures. (Skip if the fix is app-code-only.)
  • 👤 Verified live — the original failure is reproduced and confirmed fixed in a browser on production.
  • 🤖 Tagged + back-mergedv[VERSION]-[next letter] pushed; fix merged into main, develop, and every non-production branch listed in Zaj-PROJECT.md.
  • 👤 Cleanup approved — incident confirmed resolved before the hotfix branch is deleted.