3 · Harden, verify & sync (P3–P7)

TL;DR

Objective — lock down the installer-friendly 777 permissions the deployer leaves behind, confirm production schema matches staging, review ServerSync, prove production mail flow, then fast-forward every branch back to a shared base and tag the release — so you have a clean, drift-free, rollback-anchored production state.

Steps at a glance:

1. Lock down permissions and runtime caches — Tighten the installer-friendly permissions once the installer is done, then clear/warm caches against the final production .env.
2. Verify schema parity — Production must match the selected non-production launch candidate — a logical schema diff is cleanest.
3. Capture server-side changes — ServerSync — The web installer and first-run writes create files on the production server that aren’t in git yet (generated config, vendor-published assets).
4. Sync branches & tag the release — Bring every branch back to a shared base, then stamp an immutable marker.
5. Verify production mail flow — Send one real production email and confirm SPF, DKIM, and DMARC pass in the received headers.
6. Record & snapshot the release — Four closing tasks make the release reproducible and auditable: redeploy if code changed during hardening, record the release in the changelog, snapshot the schema as a versioned baseline, and capture the complete vendor-customization manifest.

User part + done-when

🤖 Agent part — almost everything: every step here is terminal-runnable over SSH and git. 👤 A person only owns credential entry, live-payment/legal choices, and investigation when a real schema difference surfaces; the agent drives non-secret production admin verification/configuration after the session exists.

Done when production storage permissions are tightened, caches are rebuilt against the final .env, migrate --pretend reports nothing pending, the non-production↔production schema diff is clean, a production test email passes SPF/DKIM/DMARC, configured branches are synced to the release branch, and the release tag is pushed.  ·  ⏱ ~1 hr  ·  🔀 mostly agent-runnable.

Background

Two short passes catch the most common day-one production failures: the deployer may leave installer-friendly 777 permissions behind, and schema drift between environments causes runtime errors and data corruption — production must match staging.

Load the production row and the validated non-production comparison row from Zaj-PROJECT.md before running checks:

PRODUCTION_BRANCH="production"
PRODUCTION_TARGET="production"
SSH_PRODUCTION_ALIAS="<production-alias-from-Zaj-PROJECT>"
NON_PROD_BRANCH="staging"          # example; selected branch from Zaj-PROJECT.md
NON_PROD_ALIAS="<non-prod-alias>"  # example; selected SSH alias from Zaj-PROJECT.md
DEPLOY_DOMAIN="<production-domain>"
DEPLOY_PATH="~/domains/$DEPLOY_DOMAIN/deploy"
# Expected: values match the production and selected non-production rows

Steps

1. Lock down permissions (P3)

Tighten the installer-friendly permissions once the installer is done.

1. Reset storage permissions and re-lock the .env.

   ssh "$SSH_PRODUCTION_ALIAS" "cd $DEPLOY_PATH/shared/storage \
     && find . -type d -exec chmod 775 {} \; && find . -type f -exec chmod 664 {} \;"
   ssh "$SSH_PRODUCTION_ALIAS" "chmod 640 $DEPLOY_PATH/shared/.env"
   # Expected: storage dirs become 775, files 664, and .env is 640

   • ✅ Storage directories are 775, files 664, and the .env is 640.

2. Clear and warm caches against the final production .env.

   PHPBIN="<production-cli-php-from-Zaj-PROJECT>"
   ssh "$SSH_PRODUCTION_ALIAS" "cd $DEPLOY_PATH/current && \
     $PHPBIN artisan optimize:clear && \
     $PHPBIN artisan config:cache && \
     $PHPBIN artisan route:cache && \
     $PHPBIN artisan view:cache"
   ssh "$SSH_PRODUCTION_ALIAS" "cd $DEPLOY_PATH/current && \
     $PHPBIN artisan about --no-ansi | grep -iE 'environment|debug|cache'"
   # Expected: caches clear/warm successfully; about reports production and debug false

   • ✅ Production runtime reads the final .env, caches are rebuilt, and debug is false.

2. Verify schema parity (P3)

Production must match the selected non-production launch candidate — a logical schema diff is cleanest.

1. Check pending migrations — expect Nothing to migrate (or only those explicitly left pending).

   ssh "$SSH_PRODUCTION_ALIAS" "cd $DEPLOY_PATH/current && php artisan migrate --pretend"
   # Expected: "Nothing to migrate" (or only intentionally-pending migrations)

   • ✅ No unexpected pending migrations.

2. Compare the selected non-production schema vs production. A logical schema diff is cleanest; a filtered migrate:status/dump diff works without extra tooling. If real differences appear, stop and investigate before continuing.

   • ✅ The selected non-production↔production schema diff is clean (or every difference is explained and resolved).

3. Sanity-check vendor migration file counts — and triage a mismatch, don’t panic. Compare the count of vendor migration files on each host; a mismatch is often expected, not a fault.

   NON_PROD_COUNT=$(ssh "$NON_PROD_ALIAS" "cd ~/domains/<non-production-domain>/deploy/current \
     && find vendor -name '*.php' -path '*/database/migrations/*' 2>/dev/null | wc -l | tr -d ' '")
   PROD_COUNT=$(ssh "$SSH_PRODUCTION_ALIAS" "cd $DEPLOY_PATH/current \
     && find vendor -name '*.php' -path '*/database/migrations/*' 2>/dev/null | wc -l | tr -d ' '")
   echo "Non-production: $NON_PROD_COUNT | Production: $PROD_COUNT"
   [ "$NON_PROD_COUNT" = "$PROD_COUNT" ] && echo "Match" || echo "Mismatch — triage below"
   # Expected: "Match", OR a mismatch you can explain via the table

   Situation	Cause	Action
   Production has more files	Stale composer cache on the server	dep clear_composer_cache production, then redeploy
   Selected non-production target has more, diff is only dev packages (debugbar, telescope, dusk)	Dev deps not deployed under --no-dev	Expected and correct — no action
   Selected non-production target has more, diff shows non-dev packages	Deploy didn’t finish cleanly	Redeploy: dep deploy production

   • ✅ Counts match, or the difference is explained by the table (dev-only packages absent under --no-dev is expected).

4. Export the production schema baseline into your 6-Schema folder for future diffs.

   • ✅ A production schema baseline is saved for future comparison.

Dump noise ≠ schema difference

Raw SQL dumps include server-version headers, AUTO_INCREMENT values, and charset notation that vary between hosts. Filter those out before judging a diff — but never wave away a real table/column/index change as “just a newline”.

3. Capture server-side changes — ServerSync (P4.5)

The web installer and first-run writes create files on the production server that aren’t in git yet (generated config, vendor-published assets). Capture them back — but production capture is far more dangerous than staging, so review every change by hand.

Never capture storage/installed (or any per-env runtime marker) back into git

The installer-complete marker (storage/installed — or your vendor’s variant) is per-environment runtime state, the same category as .env. It lives in the server’s shared storage/ and must stay out of git. If a ServerSync PR proposes adding it (an A entry for storage/installed, .installed, public/install.lock, etc.), reject that addition — a repo-baked marker rides every future clone/deploy and makes the next fresh server’s CheckInstallation middleware skip /install, leaving an un-installed app that looks “done.”

# Before merging, confirm the PR is NOT adding the install marker (adjust to your vendor's marker name).
git diff --name-status HEAD..<serversync-branch> | grep -E '^A\s+(storage/installed|\.installed|public/install\.lock)$' \
  && echo "STOP — reject the marker addition; it must stay gitignored (Phase 2 §2)" \
  || echo "OK — no install marker is being captured into git"
# Expected: "OK — no install marker is being captured into git"

Production ServerSync is review-only — never auto-merge

On production, a ServerSync that deletes a file can wipe live state. Never auto-merge a production ServerSync PR. Review every change individually — especially every D (deleted) entry — and re-confirm the clear_paths ↔ GIT_ONLY_PATHS symmetry (step 1) before applying.

Never accept a deletion of any of these — discard it with git checkout HEAD -- <path> if it appears:

.env.example   composer.lock   package-lock.json   CLAUDE.md
deploy.php      Admin-Local/**  .github/**
.claude/        .cursor/        .agent/

The inverse — storage cruft deletions are usually correct. If a vendor bundled files under storage/ that should never have been tracked, ServerSync proposing to delete them is a legitimate cleanup — accept those. The rule is protect git-only project files, allow storage-tracked-by-accident files to go. If you can’t justify a D entry, keep the file.

1. Audit clear_paths ↔ GIT_ONLY_PATHS symmetry first — and STOP on any gap. clear_paths in deploy.php drifts over time (AI configs, docs, templates). If the production ServerSync workflow’s GIT_ONLY_PATHS fell behind, the capture will propose deleting git-only files — and on production those deletions go live. Run the audit and do not trigger the capture until it passes.

   CLEAR_PATHS=$(awk '
     /add\(.clear_paths./ { capture=1; next }
     capture && /\]\);/ { capture=0 }
     capture { print }
   ' deploy.php | grep -oE "'[^']+'" | tr -d "'" | sort -u)
   
   parse_git_only_paths() {
     awk '
       /^[[:space:]]*GIT_ONLY_PATHS:/ { capture=1; next }
       capture && /^[[:space:]]*$/ { capture=0 }
       capture && /^[[:space:]]*#/ { next }
       capture && /^[[:space:]]*[A-Z_]+:/ { capture=0 }
       capture { print }
     ' "$1" | tr ' ' '\n' | grep -v '^[[:space:]]*$' | grep -v '^>-$' | sort -u
   }
   
   PROTECTED=$(parse_git_only_paths .github/workflows/capture-production.yml)
   MISSING=$(comm -23 <(echo "$CLEAR_PATHS") <(echo "$PROTECTED"))
   if [ -n "$MISSING" ]; then
     echo "STOP — these clear_paths are NOT protected by GIT_ONLY_PATHS in capture-production.yml:"
     echo "$MISSING" | sed 's/^/  /'
     echo "Add them to GIT_ONLY_PATHS, commit, merge to main, then re-run."
   else
     echo "OK — every clear_path is protected; safe to trigger production ServerSync"
   fi
   # Expected: "OK — every clear_path is protected ..." — fix and re-run until it does

   • ✅ The audit prints OK; no clear_paths entry is unprotected (or the gap was added to GIT_ONLY_PATHS, merged, and the audit re-run until clean).

2. Run the capture against production, exactly as in Phase 5 · ServerSync capture but pointed at the production alias — it opens a PR rather than committing directly.

   • ✅ A ServerSync PR is open with the production-side file changes for review (or there were no server-side changes to capture).

3. Review the diff file-by-file. Approve additions/edits that belong in git; reject anything that would delete live state or re-pull a clear_paths file. Re-run the clear_paths ↔ GIT_ONLY_PATHS symmetry audit before merging.

   • ✅ Every D (deleted) file is individually justified, the symmetry audit passes, and only intended changes merge.

4. Sync branches & tag the release (P4)

Bring every branch back to a shared base, then stamp an immutable marker.

1. Confirm no branch has diverged, then fast-forward. Check each configured branch is an ancestor of the release branch first — --ff-only will fail loudly on divergence, but the precheck tells you which branch and lets you stop before touching any.

   git fetch origin
   RELEASE_BRANCH="main"     # release branch from Zaj-PROJECT.md
   BRANCHES="$RELEASE_BRANCH develop $NON_PROD_BRANCH"
   for B in $BRANCHES; do
     git merge-base --is-ancestor "$B" "$RELEASE_BRANCH" \
       && echo "$B OK — fast-forwardable" \
       || echo "$B DIVERGED — investigate before syncing"
   done
   # Expected: every branch prints "OK — fast-forwardable"; STOP on any "DIVERGED"

   Only once each configured branch reports OK, fast-forward each non-release branch and push:

   git checkout "$NON_PROD_BRANCH" && git merge "$RELEASE_BRANCH" --ff-only && git push origin "$NON_PROD_BRANCH"
   git checkout develop   && git merge "$RELEASE_BRANCH" --ff-only && git push origin develop
   # Expected: each configured branch fast-forwards to the release branch and pushes cleanly

   • ✅ The precheck shows every configured branch OK, and develop plus the selected non-production branch point at the release branch.

2. Tag the release on the release branch and push the tag — this is your rollback and audit anchor.

   git checkout "$RELEASE_BRANCH"
   git tag -a v${VERSION} -m "Release v${VERSION}"
   git push origin v${VERSION}
   # Expected: the annotated tag is created and pushed to the remote

   • ✅ v${VERSION} is tagged on the release branch and pushed.

3. Finalize. Re-enable migrations in the deploy config for future deploys, update project status/docs to “live”, and return to develop with a clean tree.

   • ✅ Migrations re-enabled, docs marked “live”, and develop is checked out with a clean tree.

Reconfigure or prove the copied production state

If you used the fresh installer, the production admin panel is empty — redo the superadmin setup (branding, SMTP, plans, legal pages) on production, this time with live payment keys and production credentials. The agent drives non-secret panel fields and browser verification after the human login/secret moments.

If you copied a verified non-production DB, this is not a silent skip. Mark the admin setup VERIFIED-STATE only after evidence proves the copied state is production-safe:

• source DB snapshot/copy command recorded in Zaj-PROJECT.md;
• bounded stored-content search finds no selected non-production hostnames, URLs, or test credentials;
• live payment/mail/support/legal settings are production values or have owner/trigger rows in Zaj-BACKLOG.md;
• Rule 34 production seed-surface sweep below passes.

If production throws DecryptException after a non-production DB copy, start with HTTP error triage and assume an encrypted column crossed an APP_KEY boundary until proven otherwise. Use the encrypted-cast skip list from the Phase 12 cross-environment copy guidance and re-enter those values under production’s key; do not blindly rotate APP_KEY.

Error	Likely cause	Correct fix
DecryptException after copy	Staging-encrypted row copied into production	Skip encrypted-cast tables/columns, then re-enter values in production
Same error without a copy	Possible key mismatch or compromise	Investigate APP_KEY history before any rotation

Re-run Rule 34 on production before opening registration

🔴 Re-run the vendor seed-surface sweep on the PRODUCTION DB before opening registration — a staging pass does not count (Rule 34: staging ≠ prod). With the fresh installer (Strategy A) the prod DB has a new default admin@example.com → rotate it, confirm no demo users/content, and re-run the vendor-integrity / license-beacon scan on prod. Any survivor = do not open registration, regardless of the staging verdict.

5. Verify production mail flow (P6)

Phase 4 proves DNS records resolve. Production proves the app can actually send through the live provider.

1. Send one real production test email and inspect the received headers.

   # Use the app's password-reset, notification, or admin test-mail path.
   # Then inspect the received message headers in the destination mailbox or deliverability tool.
   grep -iE 'spf=pass|dkim=pass|dmarc=pass' /tmp/production-mail-headers.txt
   # Expected: headers include spf=pass, dkim=pass, and dmarc=pass

   • ✅ The received production email shows spf=pass, dkim=pass, and dmarc=pass.

If any row fails, fix the DNS/provider alignment before opening registration or sending customer mail.

6. Record & snapshot the release (P7)

Four closing tasks make the release reproducible and auditable: redeploy if code changed during hardening, record the release in the changelog, snapshot the schema as a versioned baseline, and capture the complete vendor-customization manifest.

1. Redeploy only if code changed since the P2 production deploy.

   RELEASE_BRANCH="main" # release branch from Zaj-PROJECT.md
   git log "$RELEASE_BRANCH"..develop --oneline -- "*.php" "*.js" "*.css" "*.blade.php" "deploy.php" "composer.json" "composer.lock"
   # Empty   → nothing changed; the branch sync in step 4 is enough.
   # Commits → deploy through the chain: develop → selected non-production target → release branch → production deploy target.

   • ✅ Output is empty (no redeploy needed), or the changed code was deployed through the selected non-production target → release branch → production deploy target and the release branch was re-synced.

2. Record the release. On develop, move [Unreleased] items into a new dated [vX.Y.Z] section of Zaj-CHANGELOG.md (categories: Added, Changed, Fixed, Security, Configured), and set Zaj-PROJECT.md to Production: Live · Current Release: vX.Y.Z.

   • ✅ Zaj-CHANGELOG.md has a dated [vX.Y.Z] section and Zaj-PROJECT.md reads “Live”.

3. Export a versioned schema baseline so future vendor-update diffs compare against a known-good production schema.

   Never pass production credentials on the CLI (-u 'mysql://user:pass@…' lands in shell history and ps). Use the same gitignored env pattern as Phase 5 · Atlas Cloud:

   ssh -L 3308:127.0.0.1:3306 <production-alias> -N &
   # .env.atlas (gitignored — see Phase 3 optional Atlas): ATLAS_PRODUCTION_URL="mysql://USER:PASS@127.0.0.1:3308/DBNAME"
   # Load without printing: direnv allow   OR   set -a && source .env.atlas && set +a
   [ -n "$ATLAS_PRODUCTION_URL" ] || { echo "ABORT: set ATLAS_PRODUCTION_URL in .env.atlas (never on the command line)" >&2; exit 1; }
   
   mkdir -p Admin-Local/1-Project/6-Schema/2-Snapshots/v${VERSION}-release
   atlas schema inspect -u "$ATLAS_PRODUCTION_URL" --format '{{ sql . }}' \
     > Admin-Local/1-Project/6-Schema/2-Snapshots/v${VERSION}-release/production.sql
   # Non-Atlas alternative: mysqldump --no-data via the same SSH tunnel + ~/.my.cnf (Phase 4)

   • ✅ A 2-Snapshots/v${VERSION}-release/production.sql baseline is saved (and copied into 1-Current/).

4. Capture the vendor-customization manifest — the complete list of every app file you changed from the pristine vendor. On a first full setup this is the capstone record: it’s your vendor-update playbook later (these are the files a new vendor release can conflict with) and the audit trail of your whole customization surface. vendor/ is gitignored, so this is the app-file diff against the frozen author-v${VERSION} baseline created in Phase 2.

   mkdir -p Admin-Local/3-Versions/1-v${VERSION}/2-Modifications
   git diff --name-status "author-v${VERSION}..HEAD" \
     > Admin-Local/3-Versions/1-v${VERSION}/2-Modifications/vendor-drift-v${VERSION}.txt
   # Expected: a list of A/M/D paths — the full set of changes from the vendor original
   echo "Modified vendor files — each MUST be recorded in Zaj-CUSTOMIZATIONS.md:"
   git diff --name-status "author-v${VERSION}..HEAD" \
     | awk '$1 ~ /^M/ {print $2}' \
     | grep -vE '^(packages/ZajModules/|resources/vendor-customizations/|Admin-Local/)' \
     || echo "  (none — every edit is inside a customization dir)"

   • ✅ A 2-Modifications/vendor-drift-v${VERSION}.txt manifest is saved, and every modified vendor file it lists is recorded in Zaj-CUSTOMIZATIONS.md (or wrapped in ZAJ:BEGIN/END markers).

Checklist

Do not mark this step done until every box below is checked.

• 🤖 Permissions + caches locked — storage 775/664, .env 640, production caches rebuilt, and runtime proof shows debug false.
• 🤖 No pending migrations — migrate --pretend reports nothing unexpected.
• 🤖 Schema parity verified — selected non-production↔production diff clean, production baseline exported.
• 🔀 ServerSync reviewed (P4.5) — production ServerSync PR reviewed file-by-file (no blind deletes), symmetry audit passed, and confirmed the PR does not add storage/installed (or the vendor’s install marker) into git — or confirmed there were no server-side changes to capture.
• 🤖 Branches synced — develop and every configured non-production branch fast-forwarded to the release branch (main in setup-new).
• 🤖 Release tagged — v${VERSION} annotated on the release branch and pushed.
• 🔀 Production mail proof passed (P6) — a real production test email shows spf=pass, dkim=pass, and dmarc=pass.
• 🔀 Production admin state proved (P6) — fresh-installer admin setup was redone with production values, or copied-DB admin setup is VERIFIED-STATE with source snapshot, non-production URL/search proof, live-provider status, and Rule 34 evidence.
• 🤖 Redeploy check (P7) — git log main..develop is empty, or the changed code was redeployed through the selected non-production target → release branch → production deploy target.
• 🤖 Release recorded (P7) — Zaj-CHANGELOG.md has a dated [vX.Y.Z] section and Zaj-PROJECT.md reads “Live”.
• 🤖 Schema baseline snapshotted (P7) — 2-Snapshots/v${VERSION}-release/production.sql exported.
• 🤖 Vendor-drift manifest captured (P7) — vendor-drift-v${VERSION}.txt saved; every modified vendor file is recorded in Zaj-CUSTOMIZATIONS.md.

→Next step

Monitor & day-2