Triage Runbook - Top-Down Isolation Before Any Fix

Triage Runbook - Top-Down Isolation Before Any Fix A workflow diagram generated by Archify. 01 / Isolate, top down 02 / Hand off to the domain skill EX / Known traps Who owns the name Edge + path Service + app Symptom · "the panel is down" · Isolate, top down › Who owns the name Symptom "the panel is down" Which hostname? · CT211 or CT111 owns it · Isolate, top down › Who owns the name · split front door Which hostname? CT211 or CT111 owns it split front door Resolve it · from the app host · Isolate, top down › Who owns the name Resolve it from the app host Edge or origin? · 403, no nginx log · Isolate, top down › Edge + path Edge or origin? 403, no nginx log Firewall path · ACCEPT above the DROP? · Isolate, top down › Edge + path Firewall path ACCEPT above the DROP? Service, then app · unit state, then logs · Isolate, top down › Service + app Service, then app unit state, then logs dns-mail-web · DNS · TLS · proxy · Hand off to the domain skill › Who owns the name dns-mail-web DNS · TLS · proxy network-firewall · DNAT · FORWARD · mesh · Hand off to the domain skill › Edge + path network-firewall DNAT · FORWARD · mesh game-hosting · panel · wings · wake · Hand off to the domain skill › Service + app game-hosting panel · wings · wake sites-enabled is dead · vhosts are inline · Known traps › Edge + path sites-enabled is dead vhosts are inline Cloudflare caches · old assets survive · Known traps › Service + app Cloudflare caches old assets survive wrong answer no ACCEPT wings or panel name it before editing a vhost before believing a deploy Legend Agent logic Policy Cloud service External system

Why top down, every time

  • • The front door is split: panel and billing terminate in NYC, status and webmail in LAX
  • • A valid route proves nothing - the inter-VLAN catch-all DROP has been the answer four separate times
  • • ICMP passing proves nothing either; only filter/FORWARD read in order does
  • • Resolve names from the host that owns the app: several are split-horizon

Changes that report success and do nothing

  • • Editing sites-enabled on either proxy: nginx -t passes, Ansible says changed, the change is inert
  • • Cloudflare keeps serving the old asset after a clean deploy - bump a version query and verify through the public hostname
  • • A panel config push rewrites wings config.yml but does not restart wings, so binding changes apply on the next restart
  • • blueprint -install reports SUCCESS even when the webpack build failed and stripped the assets

House rules before acting

  • • Never restart or reload a service unless explicitly told to
  • • Always run a playbook with --check first
  • • Reach every host through Ansible; local pct cannot see remote-node guests
  • • Consider the live Minecraft servers before any infrastructure change