7 · ServerSync capture

TL;DR

Objective — run the first live ServerSync capture after the selected non-production deploy, pulling server-side changes the web installer made (config values, storage markers, generated files) back into Git as a reviewable PR — using the actual shipped capture workflow (capture-staging.yml by default) so the next environment starts from a complete picture.

Steps at a glance:

1. Verify the workflow is available — GitHub surfaces workflow definitions from the default branch. In this playbook that branch is develop; main is the release branch.
2. Version parity: is the default branch workflow current? — If you edited the workflow on the selected source branch (added GIT_ONLY_PATHS, changed CODE_EXCLUDES) but haven’t merged to the default branch, running without --ref uses the default branch’s stale copy — which can propose deleting protected files.
3. Re-audit clear_paths ↔ GIT_ONLY_PATHS — This audit prevents ServerSync from proposing the deletion of files that should only exist in Git.
4. Trigger the workflow and watch it — Run the workflow and follow it to completion.
5. Review the PR file-by-file — Start from a clean local tree, then stage the PR changes without committing so you can accept/discard per file.

User part + done-when

👤 User part — almost none: this step is gh CLI + git, fully terminal-runnable. A person only needs to make the accept/discard call on ambiguous files during the file-by-file PR review.

Done when the capture workflow is available on the GitHub default branch or explicitly run with --ref "$SOURCE_BRANCH", the clear_paths ↔ GIT_ONLY_PATHS audit is clean, the run succeeds, and the PR is reviewed file-by-file with protected files preserved.  ·  ⏱ ~20 min  ·  🤖 agent-runnable.

Background

The web installer creates files on the server (config values, storage markers, generated assets) that aren’t in your repo. Phase 4 authored the ServerSync workflow and secrets; this page is the first [RUN-LIVE] capture proof after the selected non-production deploy. Read Zaj-PROJECT.md first and set SOURCE_BRANCH to the selected non-production environment row’s source branch (develop, qa, uat, etc.). The shipped workflow filenames are capture-staging.yml and capture-production.yml; do not invent old ServerSync filenames.

Where this page sits in the phase:

🟡 SHOULD — recommended, not a phase gate

The Phase 5 gate (page 6) covers every 🔴 MUST. This page and pages 8–9 are quality tracks. ServerSync is strongly recommended if your web installer writes files on the server that should live in version control — skip it only if your installer writes nothing trackable.

flowchart LR
  Install[Web installer on server] --> Files[Generated / config files]
  Files --> WF[ServerSync workflow]
  WF --> PR[PR back to git]
  PR --> Repo[Repo stays source of truth]

Steps

1. Verify the workflow is available

GitHub surfaces workflow definitions from the default branch. In this playbook that branch is usually develop; main is the release branch. Two failure modes hide here: the workflow is missing from the default branch, or the selected branch copy is stale versus the branch you are about to capture from.

1. List the workflows GitHub sees.

   REPO=$(gh repo view --json nameWithOwner -q '.nameWithOwner')
   DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef -q '.defaultBranchRef.name')
   CAPTURE_WORKFLOW=$(ls .github/workflows/capture-staging.yml 2>/dev/null || true)
   test -n "$CAPTURE_WORKFLOW" || { echo "STOP — expected .github/workflows/capture-staging.yml"; exit 1; }
   CAPTURE_WORKFLOW=$(basename "$CAPTURE_WORKFLOW")
   gh workflow list --repo "$REPO"
   # Expected: the capture workflow appears in the list; DEFAULT_BRANCH is the repo's real default branch

   • ✅ Workflows are listed (or you route to the table below to diagnose).

Read the result:

Result	Cause	Action
Workflows listed	Ready — verify parity (section 2)	Continue
Empty	File not on the default branch	Path B below
Error	CLI not authenticated	gh auth status

2. Version parity: is the default branch workflow current?

If you edited the workflow on the selected non-production branch (added GIT_ONLY_PATHS, changed CODE_EXCLUDES) but haven’t merged to the default branch, running without --ref uses the default branch’s stale copy — which can propose deleting protected files.

1. Compare the workflow blob on the default branch vs the selected source branch.

   SOURCE_BRANCH="develop" # example; use the branch from Zaj-PROJECT.md
   git fetch origin "$DEFAULT_BRANCH" "$SOURCE_BRANCH" --no-tags
   for wf in ".github/workflows/$CAPTURE_WORKFLOW"; do
     M=$(git ls-tree "origin/$DEFAULT_BRANCH" "$wf" | awk '{print $3}')
     S=$(git ls-tree "origin/$SOURCE_BRANCH" "$wf" | awk '{print $3}')
     [ -z "$M" ] && { echo "$wf: NOT ON DEFAULT → Path B"; continue; }
     [ "$M" = "$S" ] && echo "$wf: in sync (no --ref needed)" || echo "$wf: DRIFT → use --ref $SOURCE_BRANCH (Path A)"
   done
   # Expected: "in sync", "DRIFT → use --ref <branch>", or "NOT ON DEFAULT → Path B"

   • ✅ You know whether to run plain, with --ref "$SOURCE_BRANCH" (Path A), or to fix the default branch first (Path B).

Pick the path the comparison pointed at:

Path A — on the default branch but the selected branch is newer (recommended). Don’t merge the non-production branch into the release branch just to refresh a workflow mid-Phase-5 (it promotes unvalidated commits). Use --ref to run that branch’s definition directly:

1. Run the selected branch’s workflow definition directly.

   gh workflow run "$CAPTURE_WORKFLOW" --ref "$SOURCE_BRANCH" --repo "$REPO"
   git show "origin/$SOURCE_BRANCH":.github/workflows/"$CAPTURE_WORKFLOW" | grep -A30 "GIT_ONLY_PATHS"
   # Expected: the workflow dispatches against the selected branch's definition

   • ✅ The run uses the selected branch’s up-to-date workflow definition.

Path B — not on the default branch at all. Cherry-pick only the workflow commits (keep release branches clean); full merge is the last resort:

1. Cherry-pick the workflow commits onto main.

   WF_COMMITS=$(git log "origin/$DEFAULT_BRANCH..origin/$SOURCE_BRANCH" --oneline -- .github/workflows/"$CAPTURE_WORKFLOW" | awk '{print $1}')
   git checkout "$DEFAULT_BRANCH"
   for sha in $(echo "$WF_COMMITS" | tac); do git cherry-pick "$sha"; done
   git push origin "$DEFAULT_BRANCH" && git checkout "$SOURCE_BRANCH"
   # Expected: the workflow now exists on the default branch; you're back on the selected branch

   • ✅ The workflow is on main and you’re back on the selected branch.

If workflows still don’t appear, lint for YAML errors: actionlint .github/workflows/*.yml.

3. Re-audit clear_paths ↔ GIT_ONLY_PATHS

This audit prevents ServerSync from proposing the deletion of files that should only exist in Git.

Do not skip this

If clear_paths in deploy.php gained entries after Phase 4 (teams add AI configs, docs, templates over time) but GIT_ONLY_PATHS in the workflow wasn’t updated to match, ServerSync will propose deleting files that should only exist in Git.

1. Diff clear_paths against GIT_ONLY_PATHS.

   CLEAR_PATHS=$(awk '/add\(.clear_paths./{c=1;next} c&&/\]\);/{c=0} c' deploy.php \
     | grep -oE "'[^']+'" | tr -d "'" | grep -vE '(^|_)strict$|clear_paths_strict' | sort -u)
   
   PROTECTED=$(awk '
     /^[[:space:]]*GIT_ONLY_PATHS:/{c=1;next}
     c && /^[[:space:]]{2}[A-Za-z_]+:/{c=0}
     c && /^[[:space:]]*-[[:space:]]/{sub(/^[[:space:]]*-[[:space:]]*/,""); print}
   ' ".github/workflows/$CAPTURE_WORKFLOW" | grep -vE '(^|_)strict$|clear_paths_strict' | sort -u)
   
   MISSING=$(comm -23 <(echo "$CLEAR_PATHS") <(echo "$PROTECTED"))
   [ -n "$MISSING" ] && { echo "STOP — unprotected paths:"; echo "$MISSING"; } || echo "All clear_paths protected"
   # Expected: "All clear_paths protected"

   • ✅ The audit prints All clear_paths protected.

If anything is missing: add it to GIT_ONLY_PATHS, commit, merge to the default branch or run with --ref "$SOURCE_BRANCH", and re-run the audit until clean.

4. Trigger the workflow and watch it

Run the workflow and follow it to completion.

1. Trigger the run and watch it.

   gh workflow run "$CAPTURE_WORKFLOW" --ref "$SOURCE_BRANCH" --repo "$REPO"   # drop --ref if section 2 said "in sync"
   sleep 5
   RUN_ID=$(gh run list --workflow="$CAPTURE_WORKFLOW" --limit=1 --json databaseId -q '.[0].databaseId' --repo "$REPO")
   gh run watch "$RUN_ID" --repo "$REPO"
   # Expected: the run completes; a PR opens (or it reports "no changes")

   • ✅ The run completes and opens a PR (or reports “no changes”).

If it fails, read the failing step (gh run view "$RUN_ID" --log-failed --repo "$REPO" | tail -60) and match the error:

Error	Cause	Fix
Permission denied (publickey,password)	SSH key mismatch (or staging workflow using the prod SSH_PRIVATE_KEY_BASE64 secret instead of STAGING_…)	Check the Actions SSH secret vs the server’s authorized key
Connection timed out	usually transient; else wrong host/port secret or runner IP blocked	Re-run first (≈half are transient); if it persists, confirm STAGING_HOST/STAGING_PORT hold raw values (IP + port, not the alias)
Host key verification failed	server host key changed	Re-run — workflow usually accepts new keys
rsync: connection unexpectedly closed	transient transfer drop	Re-run once
Workflow file not found	not on the default branch and --ref omitted or wrong	Section 2, Path A/B
Unexpected value 'workflow_dispatch'	YAML syntax error on the ref	actionlint, fix, commit, re-run

Permission denied (publickey,password) — check for the prod-key template bug first

The most common non-transient cause is a template bug: the staging workflow references the production SSH secret. An old or hand-renamed capture workflow uses secrets.SSH_PRIVATE_KEY_BASE64 where it should use secrets.STAGING_SSH_PRIVATE_KEY_BASE64. The staging key never matches the prod server’s authorized_keys, so every run fails with Permission denied.

1. Confirm the workflow uses the staging-scoped secret.

   grep -n "SSH_PRIVATE_KEY_BASE64" ".github/workflows/$CAPTURE_WORKFLOW"
   # Expected: it references STAGING_SSH_PRIVATE_KEY_BASE64, not bare SSH_PRIVATE_KEY_BASE64

   • ✅ The workflow points at STAGING_SSH_PRIVATE_KEY_BASE64.

2. If it shows the bare prod secret, fix it, commit, and get the corrected workflow onto main.

   sed -i.bak 's/secrets\.SSH_PRIVATE_KEY_BASE64/secrets.STAGING_SSH_PRIVATE_KEY_BASE64/' \
     ".github/workflows/$CAPTURE_WORKFLOW" && rm -f ".github/workflows/$CAPTURE_WORKFLOW.bak"
   git add ".github/workflows/$CAPTURE_WORKFLOW"
   git commit -m "fix(serversync): staging workflow uses STAGING_SSH_PRIVATE_KEY_BASE64"
   # Then use section 2 Path B if the workflow is missing/stale on main.

   • ✅ The staging workflow uses the scoped secret on the branch GitHub will run.

If the secret is already correct, the cause is a key/authorized_keys mismatch — re-encode the key the alias actually uses, verify it directly, then re-upload it as STAGING_SSH_PRIVATE_KEY_BASE64.

Re-run before you debug — and fall back to running it locally

Connection timed out is a TCP-never-connected failure (distinct from Permission denied, where TCP connects but the key is wrong). Shared hosts intermittently drop GitHub-runner IPs, so an immediate re-run resolves it roughly half the time — don’t start re-uploading secrets on a single failure. If it still fails after a few retries, ServerSync also runs locally against your ~/.ssh/config alias (same rsync + GIT_ONLY_PATHS logic, no runner, no secrets), which is a fully supported fallback some teams use by default.

Connection timed out — full recovery sequence

Re-running clears roughly half of these. If it survives a few retries, the runner genuinely can’t reach the host. Work the sequence below in order. The runner has no access to ~/.ssh/config, so the STAGING_* secrets must hold raw values (IP + port), never the alias name.

1. Verify the secrets exist and have the right shape (values are masked — you’re checking presence, not content).

   REPO=$(gh repo view --json nameWithOwner -q '.nameWithOwner')
   gh secret list --repo "$REPO" | grep -E "STAGING_HOST|STAGING_PORT|STAGING_USER|STAGING_DEPLOY_PATH|STAGING_SSH_PRIVATE_KEY_BASE64"
   # Expected: all five STAGING_ secrets present

   • ✅ All five STAGING_* secrets are listed.

2. Re-extract host / port / user from ~/.ssh/config and re-upload as raw values.

   CAPTURE_ALIAS="<non-prod-alias-from-Zaj-PROJECT>"
   HOST=$(awk "/^Host $CAPTURE_ALIAS\$/,/^Host /{if(\$1==\"HostName\")print \$2}" ~/.ssh/config | head -1)
   PORT=$(awk "/^Host $CAPTURE_ALIAS\$/,/^Host /{if(\$1==\"Port\")print \$2}" ~/.ssh/config | head -1)
   USER_VAL=$(awk "/^Host $CAPTURE_ALIAS\$/,/^Host /{if(\$1==\"User\")print \$2}" ~/.ssh/config | head -1)
   echo "HOST=$HOST PORT=$PORT USER=$USER_VAL"   # HOST must be the raw IP, not the alias
   
   printf '%s' "$HOST"     | gh secret set STAGING_HOST --repo "$REPO"
   printf '%s' "$PORT"     | gh secret set STAGING_PORT --repo "$REPO"
   printf '%s' "$USER_VAL" | gh secret set STAGING_USER --repo "$REPO"
   # Expected: HOST holds server IP; PORT holds raw SSH port

   • ✅ STAGING_HOST and STAGING_PORT are raw values.

3. Retry with exponential backoff.

   for attempt in 1 2 3; do
     echo "=== Attempt $attempt ==="
     gh workflow run "$CAPTURE_WORKFLOW" --ref "$SOURCE_BRANCH" --repo "$REPO"
     sleep 10
     RUN_ID=$(gh run list --workflow="$CAPTURE_WORKFLOW" --limit=1 --json databaseId -q '.[0].databaseId' --repo "$REPO")
     gh run watch "$RUN_ID" --repo "$REPO" --exit-status && break
     [ "$attempt" -lt 3 ] && sleep 60
   done
   # Expected: one attempt completes, or all three exhaust and you fall through to local fallback

   • ✅ A retry succeeds, or all three attempts fail consistently.

4. Still failing after 3 attempts → run ServerSync locally.

   bash Admin-Local/1-Project/3-Templates/Scripts/server-sync-local.sh "$CAPTURE_ALIAS"
   # Expected: the same capture branch / PR the workflow would have produced

   • ✅ The local run produces the capture without depending on GitHub runner reachability.

Timed-out vs permission-denied are different failures

Connection timed out means TCP never connected (transient drop, wrong host/port, or cloud-IP block). Permission denied means TCP connected but the key was wrong. If you see permission errors, use the SSH-key fix above.

A successful run opens a PR (or reports “no changes”). Treat it like any code review — scrutinize anything under config/, storage/, or resources/lang/ that may carry installer-generated values you don’t want committed.

5. Review the PR file-by-file

Start from a clean local tree, then stage the PR changes without committing so you can accept/discard per file.

1. Stage the PR changes without committing.

   git checkout "$SOURCE_BRANCH" && git pull origin "$SOURCE_BRANCH"
   git fetch origin pull/PR_NUMBER/head:serversync-review
   git merge serversync-review --no-commit --no-ff
   git diff --cached --name-status
   # Expected: the staged changes listed by status (A / M / D), nothing committed yet

   Before committing the captured changes, confirm the diff excludes per-env state — storage/installed, public/storage, public/packages must never be pulled back into git (the workflow’s GIT_ONLY_PATHS should already exclude them; this is the verify-at-capture-time backstop).

   • ✅ The PR changes are staged and listed by status, with nothing committed yet.

2. Discard anything that should stay in Git, then commit and close.

   git checkout HEAD -- .env.example composer.lock package-lock.json   # common keep-in-git files
   git diff --cached --name-status
   git commit -m "ServerSync: capture non-production installer changes"
   git push origin "$SOURCE_BRANCH"
   gh pr close PR_NUMBER --repo "$REPO" --comment "Reviewed and merged file-by-file"
   git branch -d serversync-review
   # Expected: wanted changes committed and pushed; protected files preserved; PR closed

   • ✅ Wanted changes are committed and pushed; protected files (.env.example, lockfiles) are preserved; the PR is closed.

Read the status column when deciding accept/discard:

Status	Meaning	Default
A	added	usually accept
M	modified	review
D	deleted	review carefully

If the PR diff is already clean end-to-end, the shortcut is gh pr merge PR_NUMBER --squash — but only after you’ve actually read it.

Checklist

Do not mark this step done until every box below is checked.

• 🤖 Workflow available — capture-staging.yml (or the derived capture workflow) available on the default branch; version parity checked (--ref "$SOURCE_BRANCH" if drift).
• 🤖 Audit clean — clear_paths ↔ GIT_ONLY_PATHS audit returns clean.
• 🔀 PR reviewed — workflow run succeeded; PR reviewed file-by-file.
• 🤖 Protected files preserved — wanted changes committed; .env.example + lockfiles preserved.

→Next step

Deep codebase audit