MANDATORY: Consult this file before writing any new installation script. Cross-reference all relevant lessons and implement fixes proactively. Do not write a script that repeats a known failure.
What happened: The server was provisioned with Ubuntu 24.04 LTS. The TacticalRMM installer explicitly rejects it: ERROR: Only Debian 11, Debian 12 and Ubuntu 22.04 are supported.
Fix: Rebuild the server with Ubuntu 22.04 LTS. The setup-server1-full skill enforces this with a pre-flight OS check.
comment inline syntax fails on Ubuntu 22.04What happened: ufw allow 80/tcp comment "TRMM - Let's Encrypt / redirect" returned ERROR: Invalid syntax. The forward slash and apostrophe in the comment string confused UFW's argument parser.
Fix: Remove inline comments from UFW rules entirely. Rules are self-explanatory by port number.
What happened: After rebuilding the server, the SSH key (xzopia_server1) was added to the xzopia user's authorized_keys but not to root. The deployment CLI targets root by default, so all SCP/SSH calls failed with Permission denied (publickey).
Fix (for future fresh installs): During provisioning, add the SSH public key to root via the hosting provider's control panel — before first boot. Alternatively, ssh-copy-id -i <key>.pub root@<ip> immediately after the server is reachable.
What happened: Running setup-server1-full --user xzopia reached sudo bash <script> which prompted for a password. The deployment CLI runs in a non-TTY environment (Claude Code's bash executor), so the password prompt hung and then failed.
Fix: Either run as root, or pre-grant passwordless sudo before invoking the CLI:
echo 'xzopia ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/xzopia
sudo chmod 440 /etc/sudoers.d/xzopia
StrictHostKeyChecking=accept-new requires a prior known_hosts entry after server rebuildWhat happened: After rebuilding the server, the host key changed. StrictHostKeyChecking=accept-new accepts new (unknown) hosts, but the old key may still be in ~/.ssh/known_hosts from the previous Ubuntu 24.04 install, causing a host key mismatch error.
Fix: Remove the stale known_hosts entry before connecting to a rebuilt server:
ssh-keygen -R 213.171.194.170
What happened: The TacticalRMM installer was pinned to commit 5bba3e80 as a supply-chain mitigation. That commit predated Ubuntu 22.04 support and the install failed.
Fix: Pin to a specific release commit only after verifying it supports the target OS. Use master during initial testing, then pin once a working commit is confirmed.
See docs/preflight-server1.md for the step-by-step pre-deployment checklist.
What happened: setup-server1-full runs apt-get upgrade in Step 1. On a freshly provisioned server this frequently includes a new kernel. Ubuntu's needrestart package detects the kernel change and triggers an automatic reboot, killing the SSH session and the running install script before TacticalRMM is installed. This happened on every attempt to install TacticalRMM.
Fix applied: Two changes to all setup scripts:
needrestart before running apt to suppress automatic reboots ($nrconf{kernelhints} = -1 and $nrconf{restart} = 'a').apt upgrade, check /var/run/reboot-required. If a reboot is still required, reboot cleanly with a message and exit — the user re-runs the install command once the server is back (apt steps are instant on re-run; install continues from UFW onwards).Rule: Always suppress needrestart auto-reboots in non-interactive deployment scripts. Always reboot before a long-running installer, never let the installer trigger it.
What happened: The TacticalRMM upstream install.sh explicitly checks for EUID == 0 and exits with "Do NOT run this script as root." Our wrapper ran via sudo bash, making EUID=0.
Fix applied: Wrapper script drops privileges using sudo -u "$SUDO_USER" -E bash "$REMOTE_SCRIPT" for the TacticalRMM installer call only. SUDO_USER is automatically set by sudo to the original calling user (xzopia). The installer file is chown'd to $SUDO_USER before the privilege drop so they can read it.
What happened: Passing TRMM_EMAIL=value sudo bash script.sh — sudo strips the environment by default, so TRMM_EMAIL was unset inside the script.
Fix applied: Email and other variables are written as export VAR=value statements prepended to the script file before SCP, rather than passed in the SSH command prefix. File-level exports survive sudo bash.
What happened: The TacticalRMM installer takes 20–40 minutes. When the SSH connection drops (broken pipe, session timeout, parent process killed), the remote bash process receives SIGHUP and terminates. This killed the install multiple times.
Fix applied: Long-running installers are launched with nohup sudo bash script.sh > /tmp/trmm-install.log 2>&1 & on the server, detaching from the SSH session entirely. Progress is monitored by SSHing back in and tailing the log file.
What happened: We set rmmdomain, apidomain, meshdomain, letsemail as exports before calling the installer. TacticalRMM's master branch installer no longer reads these env vars — it always prompts interactively. With nohup redirecting stdin to /dev/null, every read returned empty immediately, the while [[ -z "$var" ]] loop spun at full speed, thousands of iterations per second, OOM-killed the server twice.
Fix applied: The install script now installs expect and wraps the TacticalRMM installer in an expect script that pattern-matches prompts and sends answers. Domains and email are passed as $argv to the expect script — never interpolated into a shell command string. Safe to run under nohup.
Rule: Any installer that uses interactive read prompts must be driven by expect when run non-interactively. Never rely on env vars unless you have confirmed the installer checks for them.
Rule: After any SSH hardening work — changing the SSH port, updating UFW rules, or modifying sshd_config — run /security-review and specifically check for hardcoded 22 references in all deployment scripts.
Port 22 should never appear in deployment scripts once SSH hardening is complete. Any ufw allow 22/tcp, "-p", "22", or Port 22 in scripts will silently re-open port 22 on the next run, undoing the hardening.
What to grep for after every hardening change:
grep -rn "22" scripts/ src/ | grep -E "port.*22|22.*port|22/tcp|\"22\"|'22'"
Fix applied here: All UFW rules in setup scripts replaced ufw allow 22/tcp with ufw allow "${SSH_PORT}/tcp" where SSH_PORT is passed from the CLI --port flag.
What happened: We assumed 4 prompts (api, rmm, mesh, email). The installer actually has 5:
api.example.com — backend / API subdomainrmm.example.com — frontend / RMM subdomainmesh.example.com — MeshCentral subdomainexample.com or example.co.uk — root domain (no subdomain prefix)The email was sent to the root domain prompt, corrupting the install.
Fix: Added --root-domain CLI option (default: xzopiasecure.com) and a 4th sequential expect statement matching "example.com or example.co.uk".
What happened: The installer uses a Let's Encrypt DNS challenge (wildcard cert). At Step 6, certbot displays a _acme-challenge TXT record value and waits for the user to press Enter after adding it to DNS. This cannot be automated without DNS API access.
Fix: After the 5 automated prompts, the expect script uses interact to hand control to the user's terminal. The user sees the TXT record, adds it to their DNS provider, waits ~60 seconds, and presses Enter.
What happened: We were running sudo bash script.sh making EUID=0. The installer rejected this. We then tried sudo -u xzopia bash, which also set EUID=0 from the process perspective... actually EUID was set to xzopia's UID, but the parent environment and some paths behaved as root. The official docs are clear: run as the non-root user directly.
Fix: Script is designed to run AS xzopia (EUID ≠ 0). CLI changed from sudo bash to bash. Script checks EUID == 0 and fatals.
Must NOT run as root. The installer explicitly exits if EUID == 0. Run as xzopia directly, not via sudo bash.
Five prompts in exact order. The wrapper script must answer each sequentially:
api.example.com — backend / API subdomainrmm.example.com — frontend / RMM subdomainmesh.example.com — MeshCentral subdomainexample.com or example.co.uk — root domain (base domain only, no subdomain prefix)Email goes to prompt 5, not prompt 4. Sending the email address to the root domain prompt silently corrupts the install. The root domain must be the bare domain (e.g. xzopiasecure.com), not an email address.
Use sequential expect -ex statements. One per prompt, in order. The -ex flag does exact substring matching — immune to ANSI escape codes. Concurrent expect blocks with -re -nocase do not reliably match ANSI-coloured prompts.
What happened: nohup works for non-interactive processes. TacticalRMM requires an interactive terminal (PTY) for its prompts and certbot's DNS challenge. nohup redirects stdin to /dev/null, breaking interactive input.
Fix: Script self-execs inside screen using the [[ -z "${STY:-}" ]] pattern. Screen provides a persistent PTY that survives SSH drops. If disconnected: screen -r trmm reconnects. UFW and Fail2Ban not pre-configured — official docs show the installer handles its own firewall setup.
The following process successfully installed TacticalRMM on Server 1 (Ubuntu 22.04 LTS, SSH on port 2222).
xzopia) with passwordless sudoxzopia's authorized_keysrmm., api., mesh. all pointing to the server IPRun from your own terminal (requires a real TTY for the DNS TXT record step):
cd ~/xzopia-deploy
npm run dev -- server1 install-tacticalrmm \
--email simon@xzopia.com \
--root-domain xzopiasecure.com \
-i ~/.ssh/xzopia_server1 \
--user xzopia \
--port 2222
-t (real TTY required for screen and interact)screen -S trmm — survives SSH drops (screen -r trmm to reattach)expect answers the 5 prompts sequentially using -ex (exact match):
api.example.com → api.xzopiasecure.comrmm.example.com → rmm.xzopiasecure.commesh.example.com → mesh.xzopiasecure.comexample.com or example.co.uk → xzopiasecure.comsimon@xzopia.cominteract hands control to the user for Step 6_acme-challenge TXT record. Add it to your DNS provider, wait ~60 seconds, then press Enterssh -p 2222 -i ~/.ssh/xzopia_server1 xzopia@213.171.194.170
screen -r trmm
for s in rmm nginx postgresql nats celery celerybeat meshcentral; do
sudo systemctl is-active $s
done
Dashboard: https://rmm.xzopiasecure.com
Issues encountered and resolved during Server 2 setup (ITFlow · Vaultwarden · Wiki.js · BookStack · SecureAgent Portal on Ubuntu 24.04 LTS).
What happened: wikijs and bookstack were placed on named networks (wikijs-net, bookstack-net) to isolate their databases. The databases had the correct network but the app containers had no networks: key, so Docker placed them on the default bridge. The DB_HOST: wikijs-db hostname resolved to nothing — the containers could not see each other.
Fix: Add networks: [wikijs-net] to wikijs and networks: [bookstack-net] to bookstack. Both the app and its database must be on the same named network.
Rule: Any time a container references another by hostname (e.g. DB_HOST: wikijs-db), both containers must appear in the same named networks: list. Docker Compose does not auto-attach services to each other's networks.
internal: true on bridge networks silently prevents port publishingWhat happened: After adding the correct networks, port bindings "127.0.0.1:8182:3000" were present in the compose file and docker compose config showed them correctly parsed. But docker compose ps showed 3000/tcp with no localhost binding. The services were unreachable via Apache proxy.
Root cause: internal: true on a bridge network tells Docker not to create NAT/iptables rules for containers on that network — including port publishing rules. The ports: key is silently ignored.
Fix: Remove internal: true from wikijs-net and bookstack-net. The databases remain isolated because they have no ports: mappings and Docker's network isolation prevents external containers from joining compose-managed networks.
Rule: internal: true is appropriate for overlay networks with external routing. On compose bridge networks, it breaks port publishing. Use absence of ports: to isolate databases — not internal: true.
down + up — not just restartWhat happened: After fixing the compose file, running docker compose restart or docker compose up -d did not apply the new port bindings. The containers showed the old config (3000/tcp only).
Fix: Always docker compose down && docker compose up -d when changing port bindings, network assignments, or any structural config. Docker Compose recreates containers on up -d only if it detects a config change from its internal state; if the container was started before the compose file existed in git, it has no baseline to compare against.
Rule: After any ports:, networks:, or volumes: change: docker compose down && docker compose up -d. Never rely on restart or up -d alone for structural changes.
Architecture decision: ITFlow's official installer deploys its own Apache2 + MariaDB + PHP stack directly on the host. It is not containerised and cannot be proxied through Traefik or Nginx without significant additional work.
Solution: Use Apache2 (installed by ITFlow) as the single reverse proxy for all services. Docker services (Vaultwarden, Wiki.js, BookStack, SecureAgent) listen on localhost-only ports (8181–8184) and are proxied through Apache virtual hosts.
Rule: Do not install Nginx or Traefik on Server 2 alongside ITFlow. Apache2 handles all SSL termination and proxying. Any new service must be added as a new Apache virtual host.
What happened: The original script generated an Argon2id hash on the server via docker run vaultwarden/server /vaultwarden hash. This caused SSH to drop mid-install by pulling a Docker image over an active SSH session.
Fix: Generate a bcrypt hash locally in Node.js using bcryptjs (pure JS, no native compilation) before SCPing the script. Vaultwarden accepts both $2b$ bcrypt and $argon2id$ hashes for ADMIN_TOKEN. The plaintext password is stored in ~/.xzopia-deploy/server2-passwords.env; the hash goes into the server .env.
Rule: Never generate hashes or run Docker containers during the SSH install phase. Pre-compute all credentials locally before SCP.
.env filesWhat happened: A bcrypt hash $2b$12$... written to .env via a bash heredoc was read correctly by bash. However, Docker Compose's variable interpolation engine processes .env values for $VAR references when loading them for compose-file substitution. $2b, $12, etc. were expanded to empty strings, silently corrupting the ADMIN_TOKEN and preventing Vaultwarden admin access.
Fix: Escape every $ as $$ before writing to .env:
VAULTWARDEN_ADMIN_TOKEN_ESCAPED=$(python3 -c 'import sys; t=sys.stdin.readline().rstrip(); print(t.replace("$","$$"),end="")' <<< "$VAULTWARDEN_ADMIN_TOKEN")
printf 'VAULTWARDEN_ADMIN_TOKEN=%s\n' "$VAULTWARDEN_ADMIN_TOKEN_ESCAPED" >> "$APP_DIR/.env"
Docker Compose reads $$ as a literal $. This applies to ALL .env values that may contain $ — passwords, hashes, app keys.
Rule: Any secret that can contain $ (bcrypt hashes, argon2 hashes, base64-with-padding, generated passwords) must have $ escaped to $$ before writing to a Docker Compose .env file. Write it separately from the main heredoc so the escaping is explicit.
docker compose config before starting servicesWhat happened: Config issues (unset variables, malformed YAML, missing .env entries) can cause containers to start with wrong environment values or refuse to start entirely, with cryptic errors in docker compose up output.
Fix added to setup script: After writing docker-compose.yml and .env, run:
if ! docker compose config --quiet 2>&1; then
fatal "docker-compose.yml has config errors — services not started"
fi
This catches unresolved ${VAR} references, YAML syntax errors, and missing env vars before any containers are started.
Rule: docker compose config --quiet must produce no output and exit 0 before docker compose up. Any warning is a hard error.
Default login: admin@admin.com / password
Risk: BookStack starts with these credentials active on HTTPS. Anyone who reaches https://docs.xzopiasecure.com before you change the password has full admin access to all documentation.
Fix: The setup script completion message now shows an explicit warning. Change credentials at first login via Settings → Users → admin.
What happened: Wiki.js starts in setup wizard mode on first run. Until the wizard is completed at https://kb.xzopiasecure.com, the service is not functional — all pages return the setup prompt.
Steps: Visit https://kb.xzopiasecure.com, create the admin account, select the database (already configured via env vars), complete setup. The first-run wizard takes ~2 minutes.
docker compose config parses the entire stack — a broken .env value breaks all servicesWhat happened: During diagnosis, a test with a corrupted VAULTWARDEN_ADMIN_TOKEN in .env (containing bare $ signs) caused docker compose config to emit warnings for the entire stack, not just Vaultwarden. All service definitions that reference any .env variable showed interpolation warnings.
Rule: A single .env formatting error affects the entire compose stack. Always validate the complete config with docker compose config after any .env change, not just the service you edited.
Issues encountered and resolved during post-pentest hardening of server1 and server2.
What happened: The setup script used bcryptjs to generate a $2b$12$... bcrypt hash for VAULTWARDEN_ADMIN_TOKEN. Vaultwarden 2025.12.0 treated this as a plain-text token (not a hash) and showed the notice "You are using a plain text ADMIN_TOKEN". Admin login returned 401 despite the password matching the hash.
Root cause: Vaultwarden phased out bcrypt support for ADMIN_TOKEN in recent versions. Only $argon2id$ tokens are now recognised as hashed. $2b$ (bcrypt) tokens are treated as literal plain-text expected passwords.
Fix:
bcryptjs with hash-wasm (pure JS) to generate argon2id hashes locallym=19456, t=2, p=1 — Vaultwarden OWASP presetsrc/commands/setup-server2-full.ts to use hashArgon2id() helper~/.xzopia-deploy/server2-passwords.env with new argon2id tokenRule: Always use argon2id for Vaultwarden ADMIN_TOKEN. If the Vaultwarden log shows "plain text ADMIN_TOKEN", the hash format is not being recognized — regenerate with $argon2id$ format.
docker compose restart does not apply .env changes — must use down && rm && upWhat happened: After fixing a corrupt ADMIN_TOKEN in .env, running docker compose restart vaultwarden restarted the container but it still had the old token. The container environment only updates when the container is recreated.
Fix: docker compose stop <service> && docker compose rm -f <service> && docker compose up -d <service>
Rule: restart reuses the existing container spec. Any .env change requires a full container recreation. docker compose up -d alone will only recreate if it detects a config change; use explicit rm -f to force recreation.
add_header in location blocks overrides ALL server-level headers (including snippet includes)What happened: Added HSTS headers to an nginx server {} block via a snippet include. curl -I showed HSTS was absent. The issue was that the location / block had its own add_header directives (Cache-Control, Pragma). Nginx's inheritance rule: if ANY add_header is present in a location block, ALL server-level add_header directives (including snippet includes) are silently ignored for that location.
Fix: Add security headers directly to EACH location block that needs them, alongside the existing add_header directives in that block.
Rule: Never rely on server-level add_header snippets to reach a location block that has its own add_header. Either: (a) put security headers inside each location block, or (b) use the headers_more module (more_set_headers), which doesn't have this inheritance limitation.
$$ in bash heredoc expands to current process PID — use base64 transfer for pre-escaped secretsWhat happened: Trying to transfer a Docker Compose-escaped bcrypt/argon2id token (containing $$2b$$12$$...) through a bash heredoc. Even though the value came from a variable, $$ inside the heredoc was expanded by bash as the current shell's PID (e.g., 23783), corrupting the token.
Fix: Generate the escaped value locally using a single-quoted Python heredoc (<< 'PYEOF'), base64-encode the result, transfer via base64, decode on the server:
# Local
ESCAPED=$(python3 - << 'PYEOF'
t = '$2b$12$...'
print(t.replace('$', '$$'), end='')
PYEOF
)
B64=$(printf '%s' "$ESCAPED" | base64 -w0)
# Remote (in heredoc, $B64 is safe — no $ signs in base64)
DECODED=$(printf '%s' "$B64" | base64 -d)
printf 'KEY=%s\n' "$DECODED" >> .env
Rule: Never put $$-containing values into an unquoted heredoc. Always use base64 encoding or a single-quoted Python script to safely transport pre-escaped Docker Compose secrets across shell boundaries.
Header unset Server in SSL vhostsWhat happened: After setting ServerTokens Prod (which suppresses Apache's own version), the proxied nginx container still responded with Server: nginx/1.31.1 on the portal domain. Apache was forwarding the backend's Server header as-is.
Fix: Add Header unset Server to each SSL vhost (*-le-ssl.conf). This strips the backend's Server header; Apache then adds its own Server: Apache (version-free due to ServerTokens Prod).
Rule: ServerTokens Prod only affects Apache's own Server header. For reverse-proxied backends, also add Header unset Server to suppress the backend's version disclosure. Apply to SSL vhosts (where the actual content is served), not just HTTP redirect vhosts.
What happened: In setup-server1-full, UFW was configured with ufw allow "${SSH_PORT}/tcp" then immediately ufw --force enable. If SSH_PORT was empty, wrong, or the allow command silently failed, enabling UFW would block all SSH access — no way back in without the hosting console.
Fix applied: Added a safety check between the rule additions and ufw --force enable:
if ! ufw show added 2>/dev/null | grep -qF "ufw allow ${SSH_PORT}/tcp"; then
fatal "UFW: port ${SSH_PORT}/tcp not found in pending rules — aborting to prevent lock-out"
fi
ufw show added lists pending rules before the firewall is active. If the SSH port rule is absent, the script fatals before enabling UFW.
Rule: Never call ufw --force enable without first confirming the SSH port is in the rule set. Applied to both src/scripts/setup-server1-full.ts and scripts/server1/setup-server1-full.sh.
What happened: In setup-server1-full step 6, the script adds the SSH public key from SSH_PUBKEY env var, then calls set_sshd PasswordAuthentication no. If SSH_PUBKEY was empty (key file not found locally, or --identity not passed), the key injection step emits a warning but does not stop — then password auth is disabled with zero keys in authorized_keys. Result: complete lock-out.
Fix applied: Added a pre-check after key injection and before disabling password auth:
KEY_COUNT=$(grep -cE "^(ssh-|ecdsa-|sk-)" /root/.ssh/authorized_keys 2>/dev/null || echo 0)
if [[ "${KEY_COUNT}" -eq 0 ]]; then
fatal "No SSH public key in /root/.ssh/authorized_keys — refusing to disable password auth (would lock you out)"
fi
ok "SSH key pre-check: ${KEY_COUNT} key(s) in authorized_keys — safe to disable password auth"
Rule: Before disabling password auth, always verify at least one public key is present in authorized_keys. Emit a fatal error, not a warning, if none are found. Applied to both src/scripts/setup-server1-full.ts and scripts/server1/setup-server1-full.sh.
What happened: Server 2 (Ubuntu 24.04 LTS) uses systemd socket activation for SSH by default — the unit is ssh.socket, not ssh.service. When ssh.socket is active, it controls which port SSH listens on at the kernel level. Any Port directive in sshd_config is silently ignored on reboot: ssh.socket re-opens port 22 regardless.
This is the same issue as LESSONS_LEARNED.md entry for Server 4 (setup-server4-full already handles it). Server 2's setup script was missing the fix.
Fix applied: Added to setup-server2-full SSH hardening step (before any sshd_config changes):
if systemctl is-active --quiet ssh.socket 2>/dev/null; then
systemctl disable --now ssh.socket || true
systemctl enable --now ssh.service || true
ok "ssh.socket disabled, ssh.service enabled (Ubuntu 24.04 socket activation fix)"
fi
Also added checks to pre-flight-check (WARN if ssh.socket is active) and audit-server (FAIL if ssh.socket is active on Ubuntu 24.04) skills.
Rule: On Ubuntu 24.04, always disable ssh.socket and enable ssh.service before modifying sshd_config Port. Without this, the port change appears to work immediately (ssh.service reads it) but silently reverts on the next reboot.
What happened: setup-server4-full changed sshd_config Port to 2222 and reloaded SSH before UFW was installed. On a first run this is harmless — UFW isn't active yet. But on a re-run (e.g. after a partial failure), UFW is already enabled with only the explicitly allowed ports open. The script changes SSH to 2222 and reloads, then UFW enables with default deny incoming — port 2222 is blocked until UFW rules are applied, causing lock-out.
Fix: Four-step sequence:
ufw allow ${SSH_PORT}/tcp) and keep port 22 open (ufw allow 22/tcp)sshd_config Port to the new port and reload SSHufw delete allow 22/tcpAdded guard: if [[ "${SSH_PORT}" != "22" ]] around steps 1b and 4 so running with SSH_PORT=22 is a no-op.
Rule: On any script that changes the SSH port: open the new port in UFW first, confirm SSH is running on it, then close the old port. Never change the SSH port and enable UFW in the same step without this ordering.
What happened: setup-server4-full wrote a full Nginx config with ssl_certificate and ssl_certificate_key directives pointing to /etc/letsencrypt/live/... before the certificate existed. nginx -t failed because the cert files were missing, which blocked the entire step — certbot was never reached.
Fix: Three-phase Nginx setup:
nginx -t && systemctl start nginx — passes cleanly since no certs are referencedcertbot --nginx — obtains the cert and performs its own SSL wiringnginx -t now passes because certs existsystemctl reload nginxThis is the correct certbot --nginx workflow: start Nginx on HTTP only, then let certbot handle the initial SSL setup.
Rule: Never reference /etc/letsencrypt/live/... in an Nginx config before certbot has run. Always start Nginx on HTTP only, obtain the cert, then write the full HTTPS config.
/run/sshd privilege separation directory must be created after disabling ssh.socketWhat happened: When ssh.socket is active, systemd keeps /run/sshd alive as part of the socket unit. After systemctl stop ssh.socket, that directory no longer exists. Running sshd -t (config validation) immediately after stopping the socket fails with Missing privilege separation directory: /run/sshd, aborting the SSH hardening step.
Fix: After disabling ssh.socket and before sshd -t, create the directory explicitly:
mkdir -p /run/sshd
chmod 0755 /run/sshd
Added to setup-server4-full.ts and setup-server4-full.sh (Step 2, SSH hardening block).
Rule: On Ubuntu 24.04, whenever ssh.socket is stopped before sshd -t validation, always recreate /run/sshd first. The directory is ephemeral (/run is a tmpfs) and is not re-created by ssh.service until after the first successful reload.
What happened: On a fresh server provisioned via cloud-init (Hostinger, Hetzner, etc.), the xzopia user is created by cloud-init but /home/xzopia is left with ownership root:root and permissions 755. Any process running as xzopia that writes to the home directory — including pm2 (which writes to ~/.pm2/logs, ~/.pm2/pids) — fails with EACCES: permission denied. The pm2 systemd service was enabled at boot but always started dead because of this.
The pentest on server4 (2026-05-31) caught this: pm2-xzopia.service was inactive/dead, and all MCP services (NinjaOne, Xero, Giacom) were not running as a result.
Fix applied: Added to setup-server4-full (Step 2, before UFW and SSH changes):
if id "$DEPLOY_USER" &>/dev/null; then
chown -R "${DEPLOY_USER}:${DEPLOY_USER}" "/home/${DEPLOY_USER}"
chmod 750 "/home/${DEPLOY_USER}"
fi
Rule: After any server provisioning, immediately fix deploy user home directory ownership before running any deploy steps. Use 750 (not 755) — world-execute on a home directory is unnecessary and exposes directory contents to enumeration.
What happened: The Ubuntu 24.04 nginx package ships with ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3 in /etc/nginx/nginx.conf. The site-specific vhost config correctly restricts to TLSv1.2 TLSv1.3, overriding the global default for that vhost. However any future vhost that relies on the global default would silently inherit TLS 1.0/1.1. The pentest flagged this as medium severity.
Similarly, the default Ubuntu nginx config has # server_tokens off commented out — nginx version is disclosed in HTTP response headers unless explicitly disabled.
Fix applied:
# In setup-server4-full (Step 7):
sed -i 's/ssl_protocols TLSv1 TLSv1\.1 TLSv1\.2 TLSv1\.3/ssl_protocols TLSv1.2 TLSv1.3/' /etc/nginx/nginx.conf
# server_tokens already handled but confirmed still needed:
if ! grep -q "server_tokens off" /etc/nginx/nginx.conf; then
sed -i '/http {/a\ server_tokens off;' /etc/nginx/nginx.conf
fi
Rule: After installing nginx, always patch /etc/nginx/nginx.conf to set ssl_protocols TLSv1.2 TLSv1.3 and server_tokens off globally — do not rely on site-level config to compensate for weak global defaults.
What happened: Ubuntu 24.04 default sysctl values leave net.core.bpf_jit_harden = 0 (BPF JIT spraying attacks possible) and net.ipv4.conf.all.send_redirects = 1 (server will send ICMP redirects, potential MITM assist). Also net.ipv4.conf.all.log_martians = 0 means spoofed/invalid source-IP packets are silently dropped without logging. Pentest flagged these as medium severity.
Fix applied: Added kernel hardening drop-in to all server4 setup scripts:
cat > /etc/sysctl.d/99-hardening.conf << 'EOF'
net.core.bpf_jit_harden = 2
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
EOF
sysctl -p /etc/sysctl.d/99-hardening.conf
Written to /etc/sysctl.d/99-hardening.conf (not /etc/sysctl.conf) so it survives OS upgrades and is clearly attributed.
Rule: All server setup scripts must apply a sysctl hardening drop-in. Write to /etc/sysctl.d/99-hardening.conf, not directly to /etc/sysctl.conf. Minimum: disable ICMP redirects, enable BPF JIT hardening, enable martian logging.
PermitRootLogin yes was not set during server2 initial setup — SSH hardening step was missingWhat happened: Server 2's setup script had no SSH hardening step. TacticalRMM's installer handles SSH hardening on server1, but server2 has no equivalent. The pentest found PermitRootLogin yes (HIGH severity) and multiple SSH forwarding settings enabled.
Fix: Added a dedicated SSH hardening step to both setup-server2-full.ts and setup-server2-full.sh (step 5 of 9): PermitRootLogin prohibit-password, X11Forwarding no, AllowTcpForwarding no, AllowAgentForwarding no, MaxAuthTries 3, MaxSessions 2.
Rule: Every server setup script must include an explicit SSH hardening step. Never assume SSH is hardened by the OS default or by another installer.
What happened: When writing docker-compose.yml via a bash heredoc in a setup script, any ${VAR} inside the YAML was expanded by bash before the file was written — producing literal environment variable values baked into the compose file rather than Docker Compose substitution placeholders. This caused two problems: (1) the compose file contained hardcoded values that couldn't be updated by changing .env, and (2) any ${VAR} that was unset in the current shell produced empty strings silently.
Fix: Use a quoted heredoc delimiter for the compose file:
cat > "$APP_DIR/docker-compose.yml" << 'COMPOSE_EOF'
services:
vaultwarden:
environment:
DOMAIN: "https://${VAULTWARDEN_DOMAIN}" # preserved as-is — Docker Compose reads .env
COMPOSE_EOF
Single-quoting the delimiter (<< 'COMPOSE_EOF') turns off all bash expansion inside the heredoc. Docker Compose then reads .env at runtime and substitutes ${VAR} itself.
Use an unquoted heredoc for the .env file itself — that is where bash expansion is wanted:
cat > "$APP_DIR/.env" << DOTENV_EOF
VAULTWARDEN_DOMAIN="$VAULTWARDEN_DOMAIN"
DOTENV_EOF
Rule: Docker Compose YAML heredocs: always use << 'QUOTED_DELIMITER'. .env heredocs: use << UNQUOTED_DELIMITER so bash fills in the values.
What happened: Using certbot --nginx on a server where Apache is the reverse proxy (Server 2) causes certbot to look for nginx config files and fail or produce incorrect SSL config. The correct flag depends entirely on which reverse proxy is installed.
Rules:
certbot --apache -d <domain> --non-interactive --agree-tos --email simon@xzopia.com --no-eff-email --redirectcertbot --nginx -d <domain> — but only after the HTTP-only server block is already in place and nginx is running (see lesson 38)After certbot runs, always add HSTS manually (certbot does not add it). See lesson 45.
What happened: Running certbot --apache --redirect creates an SSL vhost at /etc/apache2/sites-available/<domain>-le-ssl.conf and adds a redirect from HTTP → HTTPS. It does NOT add a Strict-Transport-Security header. All five domains on Server 2 were missing HSTS after the initial certbot run.
Fix: After every certbot run on Apache, inject HSTS into the generated SSL vhost:
harden_ssl_vhost() {
local domain="$1"
local ssl_conf="/etc/apache2/sites-available/${domain}-le-ssl.conf"
if [[ -f "$ssl_conf" ]] && ! grep -q "Strict-Transport-Security" "$ssl_conf"; then
sed -i 's|</VirtualHost>| Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"\n</VirtualHost>|' "$ssl_conf"
fi
}
Then apache2ctl configtest && systemctl reload apache2.
For Nginx: add add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; to the server {} block that handles port 443.
Rule: HSTS is never added automatically by certbot regardless of which reverse proxy is used. Always add it as a separate step after certbot. Write it into setup scripts so it applies on every fresh install.
--host 127.0.0.1 to prevent binding to all interfacesWhat happened: Server 3's systemd MCP service units use ExecStart=mcp-proxy --port 3001 --stateless node dist/index.js without specifying --host. mcp-proxy defaults to binding on all interfaces (0.0.0.0:3001). UFW was the only thing preventing external access to the raw, unauthenticated MCP endpoints. During the pentest on 2026-06-01, this was rated HIGH severity because a single misplaced firewall rule would expose all three MCP endpoints publicly.
Fix: Add --host 127.0.0.1 to each MCP service unit:
ExecStart=mcp-proxy --host 127.0.0.1 --port 3001 --stateless node dist/index.js
Then systemctl daemon-reload && systemctl restart mcp-ninjaone mcp-xero mcp-giacom.
Rule: All MCP proxy services must bind to 127.0.0.1, never to 0.0.0.0. This is true regardless of firewall rules — defence in depth requires the service itself to only accept local connections.
What happened: Nmap from inside Server 3 found port 2019 open with an HTTP service (caddy admin api). Caddy's admin API is enabled by default on 127.0.0.1:2019 with no authentication. Any local process running on the server — including mcp-runner and any compromised service — can reconfigure Caddy, retrieve TLS certificates, or reload the reverse proxy configuration without any credentials.
Fix: Add admin off to the global block in /etc/caddy/Caddyfile:
{
admin off
}
Then systemctl reload caddy.
Rule: Caddy's admin API must be explicitly disabled on production servers. If you need to reload Caddy config, use systemctl reload caddy instead of the admin API.
header block explicitlyWhat happened: Server 3 (Caddy reverse proxy) was serving claudemcp.xzopiasecure.com with none of the standard security headers: no Strict-Transport-Security, no X-Frame-Options, no X-Content-Type-Options, no Referrer-Policy. The pentest on 2026-06-01 rated this HIGH severity. Unlike Apache and nginx, Caddy has no equivalent of Header always set or add_header that works globally — each site needs an explicit header block.
Fix: Add a header block inside the site block:
claudemcp.xzopiasecure.com {
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Frame-Options "SAMEORIGIN"
X-Content-Type-Options "nosniff"
Referrer-Policy "strict-origin-when-cross-origin"
Permissions-Policy "geolocation=(), microphone=(), camera=()"
}
reverse_proxy localhost:3001
}
Rule: Caddy security headers are never implicit. Every production Caddy site block must include an explicit header {} block with at minimum HSTS, X-Frame-Options, X-Content-Type-Options, and Referrer-Policy.
What happened: Server 3's fail2ban was configured via /etc/fail2ban/jail.conf. During a apt-get upgrade, the package maintainer's version of jail.conf overwrites any local changes. This silently removed the custom [sshd] ban settings.
Fix: All fail2ban configuration goes in /etc/fail2ban/jail.local. This file is never touched by package upgrades:
cat > /etc/fail2ban/jail.local << 'EOF'
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 5
ignoreip = 127.0.0.1/8
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 86400
EOF
systemctl restart fail2ban
Rule: Never edit /etc/fail2ban/jail.conf. Always create and maintain /etc/fail2ban/jail.local. Setup scripts must write jail.local, not jail.conf.
Too many authentication failures when SSH agent has many keys loaded — use IdentitiesOnly=yesWhat happened: When connecting to Server 3 from a machine with multiple SSH keys in the agent (e.g. ssh-add -l shows 5+ keys), the SSH client offers each key in turn. If the server's MaxAuthTries is set to 3 (which our hardening scripts set), the server disconnects after 3 failed key attempts — before it even tries the correct key. Error: Received disconnect from ... Too many authentication failures.
Fix: Add -o IdentitiesOnly=yes to the SSH command. This tells the SSH client to only use the identity file specified with -i and ignore the agent:
ssh -i ~/.ssh/xzopia_server1 -o IdentitiesOnly=yes -o PreferredAuthentications=publickey \
-p 2222 xzopia@<server>
All skill scripts that SSH into servers with a specified key should include IdentitiesOnly=yes.
Rule: Whenever using -i <keyfile> to SSH into a hardened server (MaxAuthTries=3), always add -o IdentitiesOnly=yes. Without it, authentication fails silently if the agent has more keys than MaxAuthTries.
What happened: The .gitignore contains reports/. Attempting to git add reports/pentest-*.md is silently ignored — the files never appear in git status. Committing without noticing this results in the report being lost or only existing locally.
Fix: Force-add the file:
git add -f reports/pentest-server3-2026-06-01.md
git commit -m "docs(security): add server3 pentest audit report 2026-06-01"
Rule: All pentest reports must be force-added with git add -f. Never assume git add reports/ worked — always check git status and confirm the file is staged.
What happened: The pre-commit hook (.githooks/pre-commit) blocks any commit that adds the Server 3 IP address (79.99.41.247) to a file. This is correct for infrastructure changes but also fires on documentation commits — such as pentest reports that record the IP as data.
Fix: Set the bypass env var before the commit:
ALLOW_SERVER3_CHANGES=1 git commit -m "docs(security): add server3 pentest audit report 2026-06-01"
Rule: The bypass is for documentation-only commits. Never use it for commits that modify scripts, config, or anything that could change how Server 3 is accessed. Always verify that the commit is genuinely documentation before bypassing.
uname -r does not reflect recent updates until rebootWhat happened: After apt-get upgrade, the Lynis audit on Server 3 (2026-06-01) showed a warning: KRNL-5830 — Reboot required to use updated kernel. uname -r reported 6.8.0-117, but dpkg -l linux-image-* showed 6.8.0-124 was installed that day. The running kernel doesn't change until reboot — the newer kernel is on disk but not in use. This means any kernel-level security patches are not active until the server is rebooted.
Rule: After any apt-get upgrade that updates the kernel (check dpkg -l linux-image-*), schedule a reboot during the next maintenance window. Do not declare a server's security posture as current until the running kernel matches the installed kernel. Lynis KRNL-5830 warning = reboot pending.
What happened: Server 6 (June 2026) was locked out twice via SSH. Both times the cause was the same: UFW was enabled before port 2222 was confirmed in both UFW and sshd_config. When UFW blocks port 22 before 2222 is open, all SSH connections are dropped (packet drop, not refused — looks like a timeout). Recovery required the Fasthosts web console (VNC) to re-allow port 22.
Correct sequence (mandatory):
ufw allow 22/tcp && ufw allow 2222/tcpsshd_config for port 2222 and validate with sshd -tufw --force enable)systemctl restart ssh)Rule: Never enable UFW before both the old and new SSH ports are in the allow list. Never remove the old port from UFW before confirming the new port works. Use scripts/setup/safe-ssh-ufw-setup.sh for all new servers — it enforces this order and blocks continuation if sshd -t fails. See also docs/new-server-preparation.md steps 13–18.
What happened: CIPP's frontend assumes Azure Static Web Apps (SWA) provide authentication via the managed /.auth/* endpoint layer. Self-hosting the Docker stack has no equivalent — /.auth/me is intercepted by the Nginx SPA fallback (returns HTML), /.auth/login/aad doesn't exist, and the entire session/principal injection system is absent. The login spinner never resolved because clientPrincipal in /.auth/me was always null.
Fix attempted (Session 10): Built a full Python3 SWA auth emulator service (PKCE OAuth2, server-side session store, /validate endpoint for nginx auth_request, x-ms-client-principal header injection). Authentication was solved — Microsoft OAuth login worked and the CIPP dashboard loaded.
Why abandoned: The auth layer being absent is solvable, but it is a signal that the deployment is fighting the platform's architecture. The Azurite and credential blockers (Lessons 56–57) made the overall self-hosted approach unviable.
Rule: CIPP is maintained and tested exclusively on Azure Static Web Apps. Any self-hosted deployment must reimplement the SWA auth layer, the Azure Storage layer, and the Azure Functions orchestration layer. The cumulative surface area makes self-hosting fragile and unsupported. Use the official Azure deployment path.
What happened: CIPP's backend uses Azure Durable Functions, which uses the Azure Storage SDK to manage orchestration state (blob containers, queues, tables). Azurite 3.35.0 (from the official CIPP docker-compose.yml) rejected API version 2026-02-06 with InvalidHeaderValue: The API version 2026-02-06 is not supported by Azurite. Please upgrade Azurite to latest version. The cippapi container was unable to create its Durable Task hub containers and failed to start the orchestration worker.
Root cause: CIPP's Azure SDK (DurableTask.AzureStorage) is updated frequently; Azurite lags behind. The --skipApiVersionCheck flag exists but only helps for older API calls — it does not make Azurite implement newer APIs.
Rule: Azurite is suitable for local development and simple blob/queue/table use. For production workloads using Azure Durable Functions (like CIPP), Azurite's API version lag makes it an unreliable stand-in. Use real Azure Storage (consumption pricing at Xzopia scale is ~£0.50/month — negligible).
What happened: CIPP stores Microsoft credentials (TenantID, ApplicationID, ApplicationSecret, RefreshToken) in Azure Storage, populated by its own SAM setup wizard. Attempting to manually inject credentials into /opt/cipp/.env and the Azurite tables resulted in:
AADSTS7000215: Invalid client secret provided — Microsoft rejected the ApplicationSecret even when the format was correct (40-char value, not a GUID/ID).Could not get Refreshtoken from environment variable. Reloading token. — RefreshToken was 1580 chars vs the ~1842-char token from a proper OAuth flow (likely truncated during manual transfer).Rule: CIPP credentials must be obtained and stored via the CIPP SAM wizard, not injected manually. The wizard's OAuth flow generates a correctly-formed and correctly-scoped RefreshToken. Manual injection is fragile — use the supported path (the Azure deployment wizard).
What happened: Self-hosting CIPP required reimplementing: (1) Azure SWA auth layer (/.auth/* endpoints, PKCE OAuth, session management, principal injection); (2) Azure Storage (Azurite as stand-in, which failed); (3) Azure Functions runtime (Docker). Items 1 and 3 were solved; item 2 failed in production due to API version lag.
Principle: When a deployment requires rebuilding the platform's managed layers by hand, that is a signal the architecture is wrong — stop and use the supported deployment path. The engineering cost of maintaining custom reimplementations of managed cloud services is never worth the avoided cloud cost at MSP scale.
Rule: Evaluate the supported deployment first. For CIPP: the official Azure deployment (SWA + Functions + Key Vault + Storage) costs ~£1–3/month at Xzopia's scale (consumption pricing, 70 clients, internal tool). The self-hosted attempt cost significantly more in engineering time. See docs/cipp-azure-deployment.md.
What happened: During the CIPP credential setup attempt on Server 6 (77.68.127.221), AllowTcpForwarding yes was set in /etc/ssh/sshd_config to allow SSH port forwarding for token bootstrap. This setting was not reverted before the self-hosted work was abandoned.
Fix required before next use of Server 6:
sudo sed -i 's/AllowTcpForwarding yes/AllowTcpForwarding no/' /etc/ssh/sshd_config
sudo sshd -t && sudo systemctl reload ssh
Rule: Any temporary SSH configuration changes made for debugging or bootstrap must be explicitly reverted and verified before the session ends. AllowTcpForwarding should always be no on production servers — it enables SSH tunnelling which can be used to bypass firewall rules.
What happened: The FusionPBX Debian installer (debian/resources/iptables.sh) removes UFW entirely and installs iptables-persistent with hardcoded rules. One of those rules is iptables -A INPUT -p tcp --dport 22 -j ACCEPT. Since Xzopia servers use SSH on port 2222, running the installer unpatched results in a complete SSH lockout — port 22 is opened (unused) and port 2222 is blocked by the iptables -P INPUT DROP default policy.
Fix: Before running the installer, patch the SSH port:
sudo sed -i 's|iptables -A INPUT -p tcp --dport 22 -j ACCEPT|iptables -A INPUT -p tcp --dport 2222 -j ACCEPT|g' \
/usr/src/fusionpbx-install.sh/debian/resources/iptables.sh
Rule: Before running any third-party installer that manages firewall rules, check for hardcoded port 22 references and patch them to the actual SSH port. This applies to FusionPBX, any cPanel/WHM installer, and any other tool that writes its own iptables/nftables rules.
What happened: The FusionPBX Debian installer explicitly runs ufw reset, ufw disable, apt-get remove -y ufw, and purges all UFW iptables chains. It then installs iptables + iptables-persistent with its own VoIP-optimised rule set. After installation, UFW is no longer present on the server.
What this means: Post-FusionPBX, all firewall management is via iptables/iptables-save/netfilter-persistent save — not ufw. The /audit-server skill's UFW checks will fail but that is expected. Use sudo iptables -L -n -v and sudo iptables -S to inspect rules.
Rule: After FusionPBX installation, verify iptables rules directly. Do not expect UFW to be present. Update any monitoring scripts or audit checks to accommodate iptables-based firewall.
What happened: iptables.sh runs ufw reset, which prompts "Resetting all rules to installed defaults. Proceed with operation (y|n)?". When running inside a detached screen -dmS session (no interactive stdin connected), this prompt blocks indefinitely — the installer hangs at 217 log lines.
Fix: After starting the installer in detached screen, check log progress. If stuck at the UFW reset prompt, send y via:
sudo screen -S fusionpbx -X stuff 'y\n'
Rule: When running the FusionPBX installer inside a detached screen session, monitor the log for the UFW reset prompt and respond within ~60 seconds. Alternatively, patch iptables.sh to use ufw --force reset before running.
Better fix for future: Patch the installer's iptables.sh to use ufw --force reset (non-interactive):
sudo sed -i 's/^ufw reset$/ufw --force reset/' /usr/src/fusionpbx-install.sh/debian/resources/iptables.sh
What happened: The FusionPBX debian/install.sh installs snmpd and writes rocommunity public to /etc/snmp/snmpd.conf. The public community string is the well-known default that automated scanners and attackers probe first. Any pentester running snmpwalk -v1 -c public <ip> would have read access to all SNMP OIDs — including network config, process list, and mounted filesystems.
Fix applied (patch before running):
# Remove SNMP block entirely from install.sh
sudo sed -i '/^#SNMP/,/^service snmpd restart/d' \
/usr/src/fusionpbx-install.sh/debian/install.sh
If SNMP is required later (e.g., for monitoring integration), install it manually with a strong community string or SNMPv3.
Rule: Always read and audit third-party installer scripts for default/insecure credentials before running them. The FusionPBX SNMP block must be removed or patched on every fresh install.
What happened: When planning the FusionPBX deployment, we assumed certbot (Let's Encrypt) was called during the main debian/install.sh. This would create a dependency on DNS resolving to the server during installation. In fact, the installer generates a self-signed cert for Nginx and PostgreSQL — the resources/letsencrypt.sh script is entirely separate and must be run manually after the main install.
Implication: The FusionPBX main installer has NO DNS dependency. It can be run before DNS is propagated. certbot/Let's Encrypt is applied as a separate post-install step.
Rule: Run the FusionPBX main installer independent of DNS. Fix DNS and run the SSL step (resources/letsencrypt.sh or certbot --nginx) only after the main install completes. Prefer certbot --nginx for consistency with Lessons #38/#44/#45.
What happened: The FusionPBX installer config.sh has a switch_token= field for a SignalWire Personal Access Token (PAT). Based on the field name, it appeared that source compilation would require this token. In fact, FreeSWITCH 1.10.12 with switch_branch=stable clones from https://github.com/fusionpbx/freeswitch (the FusionPBX fork, not the SignalWire repo) and explicitly disables mod_signalwire during compilation. No token is needed.
Rule: The switch_token= in FusionPBX config.sh is only required if mod_signalwire is enabled or if using SignalWire's private binary packages. For the fusionpbx/freeswitch fork with switch_version=1.10.12 and switch_source=true, leave switch_token= empty.
What happened: The FusionPBX installer at /usr/src/fusionpbx-install.sh/debian/install.sh writes its own fail2ban jail.local containing FusionPBX-specific jails (freeswitch, fusionpbx, nginx-dos, etc.). This completely overwrites any pre-hardening jail.local. After reboot, fail2ban failed with "Have not found any log file for sshd jail" because:
[ssh] jail had port = 22 (not 2222)defaults-debian.conf [sshd] jail also used port 22Fix: After the FusionPBX installer completes, create /etc/fail2ban/jail.d/sshd-override.conf to override the sshd jail port to 2222, and patch the [ssh] jail in jail.local to use port 2222.
Rule for future installs: Do NOT write a custom jail.local before running the FusionPBX installer — it will be overwritten. Instead, apply port corrections post-install via jail.d override files. The setup-server5 skill handles this in the post-install phase.
# Apply after FusionPBX installer completes:
sudo bash -c 'cat > /etc/fail2ban/jail.d/sshd-override.conf << EOF
[sshd]
enabled = true
port = 2222
filter = sshd
backend = auto
logpath = /var/log/auth.log
maxretry = 5
findtime = 300
bantime = 86400
EOF'
sudo sed -i '/^\[ssh\]/,/^\[/ s/^port.*= *22$/port = 2222/' /etc/fail2ban/jail.local
sudo systemctl restart fail2ban
What happened: After the FusionPBX install, /etc/iptables/rules.v4 contained -A INPUT -p udp -m udp --dport 1194 -j ACCEPT. OpenVPN was not installed and is not part of the FusionPBX stack. The rule is an artefact of the FusionPBX iptables template and serves no purpose.
Fix:
sudo iptables -D INPUT -p udp --dport 1194 -j ACCEPT
sudo netfilter-persistent save
Rule: Always audit the full iptables INPUT ruleset after the FusionPBX installer completes. Look for unexpected open ports (1194, any non-VoIP service). Close anything that isn't SSH, HTTP/HTTPS, SIP, RTP, or ICMP.
What happened: FreeSWITCH defaults to 16384–32768 for RTP media ports. The FusionPBX installer leaves these commented out in switch.conf.xml and opens the full 16,384-port range in iptables. For a small deployment (≤100 concurrent calls), this attack surface is unnecessarily wide.
Fix: Immediately after install, narrow the range in switch.conf.xml and update iptables:
# Edit /etc/freeswitch/autoload_configs/switch.conf.xml
# Uncomment and set:
# <param name="rtp-start-port" value="16384"/>
# <param name="rtp-end-port" value="16584"/> ← 200 ports = 100 concurrent calls
# Replace iptables rule
sudo iptables -D INPUT -p udp --dport 16384:32768 -j ACCEPT 2>/dev/null || true
sudo iptables -A INPUT -p udp --dport 16384:16584 -j ACCEPT
# Update DSCP QoS rule (mangle table) to match
sudo iptables -t mangle -D OUTPUT -p udp --sport 16384:32768 -j DSCP --set-dscp 0x2e 2>/dev/null || true
sudo iptables -t mangle -A OUTPUT -p udp --sport 16384:16584 -j DSCP --set-dscp 0x2e
sudo netfilter-persistent save
sudo systemctl reload freeswitch
Rule: 200 RTP ports = 100 concurrent calls. Scale to 400 (200 calls) if needed. Always align the iptables RTP rule, the mangle DSCP rule, and switch.conf.xml — they must all use the same range.
server_name fusionpbx — must update to domain before certbotWhat happened: The FusionPBX installer sets server_name fusionpbx; (the system hostname) in all nginx server blocks. certbot --nginx -d pbx.xzopiasecure.com needs a server block with a matching server_name to find the right vhost. Without updating it first, certbot either fails to find the vhost or picks the wrong one.
Fix: Before running certbot, update the server_name in the public-facing server blocks (port 80 and port 443):
sudo sed -i 's/server_name fusionpbx;/server_name pbx.xzopiasecure.com;/g' \
/etc/nginx/sites-available/fusionpbx
sudo nginx -t && sudo systemctl reload nginx
Leave the listen 127.0.0.1:80 localhost block unchanged (its server_name 127.0.0.1 is correct).
Rule: Always update server_name to the real domain immediately after the FusionPBX install, before certbot, DNS checks, or any curl verification.
What happened: Scripts and lessons referencing chown -R freeswitch:freeswitch /etc/freeswitch/tls fail silently because there is no freeswitch system user. FusionPBX's Debian installer runs FreeSWITCH as www-data. The /etc/freeswitch/tls directory is owned by www-data:www-data.
Fix: Use www-data for all FreeSWITCH file ownership on this installation:
sudo chown -R www-data:www-data /etc/freeswitch/tls/
Rule: Before writing scripts that chown to freeswitch, confirm the actual process user: ps aux | grep freeswitch | awk '{print $1}'. On FusionPBX/Debian installs it is typically www-data. Update all hooks and scripts accordingly.
What happened: After patching nginx.conf to add server_tokens off and adding add_header Strict-Transport-Security to the SSL vhost, systemctl reload nginx was used. The changes did not take effect: the Server: header still showed nginx/1.22.1 and the HSTS header was absent.
Fix: Use systemctl restart nginx (not reload) after any change to the http {} block in nginx.conf or after adding add_header directives to a server block for the first time.
Rule: Use reload for vhost content changes (location blocks, upstreams). Use restart for global nginx.conf changes (http {} block directives) and for new server-level add_header entries. Verify with curl -si https://domain/ | grep -E 'Server:|Strict'.
What happened: Installing msmtp on Ubuntu 24.04 via ssh host 'sudo bash -s' << 'EOF' caused the msmtp debconf question ("Enable AppArmor support? [yes/no]") to loop endlessly, consuming the entire heredoc from stdin. The script's remaining commands (creating credential files, systemd units) never ran because stdin was exhausted by the hung debconf prompt.
Fix: Always set DEBIAN_FRONTEND=noninteractive before any apt-get that installs interactive packages, and use --force-confdef/--force-confnew dpkg options to suppress all interactive prompts:
export DEBIAN_FRONTEND=noninteractive
apt-get install -y -q \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confnew" \
msmtp ...
Additional rule: Do NOT install msmtp-mta when postfix is already present — msmtp-mta provides mail-transport-agent and will remove postfix as a conflicting package. Install only msmtp (standalone client) when postfix needs to be preserved.
Rule: Any SSH heredoc script that runs apt-get must set DEBIAN_FRONTEND=noninteractive before the install command. Otherwise, interactive debconf prompts consume the heredoc stdin and all subsequent commands in the script are silently skipped.
What happened: The apt restic package version differs between Ubuntu 24.04 (0.16.4) and Debian 12 bookworm (0.14.0). Each server uses its own restic repo in B2, so there is no direct cross-version compatibility issue for normal operations. However, if you ever attempt to access a repo from a different server or a fresh restore machine, ensure restic is the same version (or newer) than the one that created the repo.
Rule: For restore operations, always install the latest restic binary from GitHub releases (restic_x.y.z_linux_amd64.bz2) rather than relying on the apt package version. Restic is a single binary with no runtime dependencies — drop-in replacement is trivial. Keep Debian bookworm's apt version updated via restic self-update as root.
What happened: FusionPBX uses a Lua XML handler (app.lua xml_handler) to serve FreeSWITCH its sofia.conf configuration dynamically from PostgreSQL. The rendered XML is cached to /var/cache/fusionpbx/<hostname>.configuration.sofia.conf with a 1-hour TTL. After changing a global variable (internal_ssl_enable, sip_tls_version, etc.) in FusionPBX Advanced → Variables and restarting FreeSWITCH, the daemon fetched the cached pre-change render — TLS did not activate despite the variable being correctly saved in the DB.
Fix: Delete the cache file before restarting FreeSWITCH:
sudo rm /var/cache/fusionpbx/<hostname>.configuration.sofia.conf
sudo systemctl restart freeswitch
FreeSWITCH will then call the Lua handler which regenerates from the DB and writes a fresh cache entry.
Rule: Any time you change a FusionPBX SIP profile setting or global variable and expect FreeSWITCH to pick it up, always delete the sofia.conf cache file first. A reloadxml followed by sofia profile internal restart may also work intra-session, but a restart-after-cache-clear is the reliable path. Symptom of stale cache: sofia status shows the old binding (e.g. no TLS on 5061) even though the DB and variable values are correct.
Background (2026): The Shai-Hulud/Miasma supply-chain campaign targets npm package maintainers, injects malicious preinstall/postinstall hooks into legitimate packages, and uses Dune-themed GitHub repositories as dead-drop exfiltration targets. Attack vector: npm install executes injected shell scripts with the installing user's privileges before the package is usable.
Audit performed 2026-06-11 (WSL dev + Server 4 MCP dev + Server 3 live read-only):
Shai-Hulud, Miasma, _Js\b) across all node_modules: 0 hits on all three boxesesbuild's known-legitimate postinstall: node install.js (downloads platform binary) — on WSL and S3. Server 4 had zero hooks across all 10 MCP projects.Hardening applied (WSL + Server 4 only — Server 3 read-only, untouched):
npm config set ignore-scripts true
npm config get ignore-scripts # must return: true
Applied to: WSL (~/.npmrc) and Server 4 (/home/xzopia/.npmrc). Verified true on both.
Why Server 4 was safe to harden without breakage: All 10 MCP projects use only @modelcontextprotocol/sdk, express, and zod — no native modules (no node-gyp, no C++ addons). Zero legitimate postinstall scripts exist across the entire S4 dependency tree.
Caveat for future installs with native modules: ignore-scripts=true blocks ALL postinstall scripts including legitimate ones (e.g. better-sqlite3, bcrypt, canvas). If a new package needs native build scripts:
# one-off install for a specific verified package
npm install --ignore-scripts=false <package>
# or temporarily
npm config set ignore-scripts false
npm install <package>
npm config set ignore-scripts true
Always audit postinstall content of any native module before running with scripts enabled.
WSL CVE found (separate from campaign): vitest ≤3.2.5 → vite ≤6.4.1 → esbuild ≤0.24.2 (GHSA-67mh-4wv8-2f99, dev server CORS bypass). Dev-only toolchain, not production. Fix requires vitest@4.x (breaking change) — address separately.
Rule: Run npm config set ignore-scripts true on every new server or dev machine as part of the base Node.js setup. This is now part of the Server 4 hardening baseline. Check for Dune-themed repos under the GitHub org manually after any PAT rotation or suspected compromise.
What happened: Both xzopia-backup.timer units (S2 and S5) were created with systemctl enable but never systemctl enable --now, and neither box was rebooted after install. Both timers sat in state inactive (dead) with Trigger: n/a — no scheduled backup ever ran. S5 was caught 2026-06-11; S2 was caught 2026-06-12 after 3 days with no Vaultwarden/ITFlow/BookStack backup, despite the backup system having been declared complete.
Root cause: The completion gate covered: script runs manually, test restore passes, msmtp alert works, healthcheck ping confirmed. It did NOT verify systemctl status <timer> showed active (waiting) with a real Trigger: timestamp. "Ran successfully when invoked manually" was silently confused with "will run on schedule."
Fix:
sudo systemctl enable --now xzopia-backup.timer
sudo systemctl status xzopia-backup.timer # must show: active (waiting); Trigger: <real future timestamp>
New mandatory completion gate for any systemd timer-based service:
systemctl status <timer> → state is active (waiting), Trigger: line shows a real future timestampjournalctl -u <service> that it fired on the scheduled timerestic snapshots count incremented, msmtp alert sent) from the unattended runRule: systemctl enable persists the unit across reboots but does NOT start it. enable --now both persists and starts. For timers on a freshly-installed box that will not be rebooted soon, always use enable --now. Verify active (waiting) explicitly — never infer timer state from a successful manual run.
What happened: Both healthchecks.io checks were configured with Period 24h + Grace 1h = 25h total alarm window. During the timer-dead investigation, manual curl pings were used to test the wiring, which kept resetting the 25h window. A backup timer that had been dead for 3 consecutive days never triggered a healthchecks.io alarm because at least one manual ping fell within each 25h window.
Fix: Both checks tightened to Period 24h / Grace 2h. A single missed scheduled run will alarm by ~03:01 on the day it is missed (timer fires at 01:01, grace expires at 03:01 — before the working day starts). Manual pings during one-off debugging can still reset the window for that day, but two consecutive missed runs will always alarm within the same morning.
Rule: Set dead-man's switch grace windows to the minimum operationally tolerable, not the maximum. A 25h grace on a 24h timer is near-useless — any manual invocation (debugging, testing, ad-hoc run) silences an alarm for a full extra day. 2h grace is the correct default for a daily overnight backup. Document the alarm time explicitly: "if backup misses, alarm arrives by HH:MM."
Context: VoiceHost SIP trunk via SIP Convergence. TLS endpoint: secure.st.sipconvergence.co.uk (FreeSWITCH external profile, port 5061).
Problem: secure.st.sipconvergence.co.uk is round-robin DNS across three nodes (185.91.41.29, 185.91.41.30, 37.157.54.206). VoiceHost issues a node-specific auth nonce on 401 Unauthorized. FreeSWITCH's DNS round-robin sends the authenticated re-REGISTER to a different node than the one that issued the nonce → nonce mismatch → 408 Timeout. Registration never completes.
Fix: Pin the gateway Proxy and Outbound Proxy fields in FusionPBX to a single node IP (185.91.41.30:5061). Keep the From-Domain / Realm set to the hostname (secure.st.sipconvergence.co.uk) — FreeSWITCH uses the realm for TLS cert validation and digest auth realm, not for DNS resolution when a proxy IP is set. Result: clean 200 OK.
Open issue (as of 2026-06-14): FreeSWITCH reuses the nonce on periodic re-register (nc increments, same nonce). VoiceHost silently rejects → 408. Ticket #162431 open with b4b/VoiceHost. Workaround: reduce re-register interval to force fresh 401/200 cycle if timeouts recur.
Rule: For any SIP trunk behind round-robin DNS, always pin proxy to a single IP. Never rely on DNS round-robin for stateful SIP digest auth.
Problem: secure.st.sipconvergence.co.uk TLS cert is signed by Starfield Secure Certificate Authority G2 (Starfield Technologies). Documentation and prior notes referenced GoDaddy SF Root G2. Loading GoDaddy CA bundles into FreeSWITCH resulted in verify code 19 (self-signed / untrusted) and TLS handshake failure.
Fix: Download the correct Starfield chain:
# From certs.starfieldtech.com
wget https://certs.starfieldtech.com/repository/sfig2.crt
wget https://certs.starfieldtech.com/repository/sfroot-g2.crt
cat sfig2.crt sfroot-g2.crt >> /etc/freeswitch/tls/ca-bundle.crt
Rule: Always confirm the actual CA before building a trust bundle:
openssl s_client -connect secure.st.sipconvergence.co.uk:5061 </dev/null 2>&1 | grep -E 'issuer|subject'
Do not assume Starfield = GoDaddy (Starfield is a GoDaddy subsidiary but uses separate roots and intermediate CAs — they are not interchangeable in trust stores).
Symptom: SIP client (Zoiper, etc.) registration fails with 403 Forbidden even when username, domain, and password are all entered correctly. fs_cli -x "user_exists id <ext> <domain>" returns true (user found in directory), but every REGISTER attempt is rejected. sofia status shows FAILED-CALLS-IN incrementing, REGISTRATIONS 0.
Root cause: FusionPBX auto-generates extension SIP passwords that may contain special characters (# @ : / % & + = etc.). Some SIP clients and FreeSWITCH itself escape or URL-encode these characters differently when computing the MD5 digest for SIP digest authentication, so the auth hashes never match despite the password being "correct." This is not a password-entry error — the password is transmitted correctly but hashes to a different value.
Diagnosis:
fs_cli -x "user_data <ext>@<domain> param password"
If the output contains any non-alphanumeric character, that is the cause.
Fix: In FusionPBX → Accounts → Extensions, edit the extension and regenerate the Password field (the SIP auth credential — NOT the Voicemail PIN) as alphanumeric only (letters + numbers, 20+ characters for entropy). Save the extension, update the SIP client, and update Vaultwarden to match.
Prevention: When creating extensions, always set SIP passwords as alphanumeric-only from the start. Apply this to all extensions and all future client tenants. Standard: 20-character alphanumeric SIP password (e.g. openssl rand -base64 15 | tr -dc 'A-Za-z0-9' | head -c20).
Symptom: Copied a greeting WAV file directly into the mailbox storage directory (/var/lib/freeswitch/storage/voicemail/<domain>/<ext>/). The file exists on disk but does not appear in the mailbox's Greeting dropdown in FusionPBX.
Root cause: FusionPBX stores greeting metadata in the database. A greeting only appears in a mailbox's dropdown after it has been registered via the central Greetings management interface. Copying files to disk skips that registration step — the file is present but invisible to the UI and to FreeSWITCH's voicemail module.
Fix / correct workflow:
Rule: Never hand-copy greeting files into storage directories. Always upload via the central Greetings interface so the DB record is created. The per-mailbox dropdown only shows DB-registered greetings.
Context: Server hardening (see lesson #75) sets ignore-scripts=true globally in ~/.npmrc as a supply-chain mitigation. This prevents postinstall scripts from running — which is the intent for production packages. This is the hardening working as intended, not a break. Do not disable the global guard.
Problem: Claude Code (@anthropic-ai/claude-code) fetches its native platform binary via a postinstall script. With ignore-scripts=true, any npm update -g or npm install -g silently skips the postinstall — the package version bumps but the native binary is not installed/refreshed, so claude fails with native binary not installed.
Standing recovery procedure (recurs on every CC update while the guard is in place):
Option A — repair only (run the one trusted postinstall by hand):
node $(npm root -g)/@anthropic-ai/claude-code/install.cjs
# On this machine the confirmed path is:
# node /home/simonkillen/.npm-global/lib/node_modules/@anthropic-ai/claude-code/install.cjs
Option B — full reinstall, scoped exception only:
npm install -g @anthropic-ai/claude-code --ignore-scripts=false
# Do NOT use: npm config set ignore-scripts false (removes the guard permanently)
Option C — force postinstall on update only:
npm update -g @anthropic-ai/claude-code --foreground-scripts
Rule: Never permanently disable ignore-scripts. Use one of the above per-command exceptions only. Confirmed working 18 Jun 2026 after a power outage (CC 2.1.181 repaired via Option A; global ignore-scripts=true left intact).
Problem: After syncing the CIPP fork, publish_release.yml contains if: github.event.repository.fork == false — it intentionally skips on forks. Pushing to a fork never triggers the Azure Function App deployment pipeline; the API stays on the old version indefinitely.
Correct deployment path for self-hosters:
pull_request: trigger from the SWA workflow to avoid duelling deploy triggers when a sync PR is opened.Rule: After first deploying CIPP via Azure, always check Function App → Deployment Center to confirm the deployment source is wired to GitHub. Do not rely on the fork's Actions workflows to push the API — they won't.
Symptom: After deploying CIPP, tenants list is empty. "Refresh token matches key vault" check fails. Token capture in the setup wizard appears to complete but tokens are never stored.
Root cause: Running "First Setup" when the CIPP-SAM application already exists in Entra (status: Activated) attempts to create a new app registration rather than refreshing tokens against the existing one. The wizard flow is different and token capture fails silently.
Fix:
Rule: If CIPP-SAM already exists in Entra, always use "Refresh Tokens for existing application registration." First Setup is for net-new deployments only. Always use a normal (non-incognito) browser for token capture.
Context: Both ITFlow (PSA) and Twenty CRM run on Server 2. Engineers and AI agents interact with both, and it is easy to create duplicate company records or route data to the wrong system.
Rule: ITFlow is the system-of-record for managed (paying) clients — company record, operational contacts, assets/devices, contracts, SLAs, tickets, invoices. Twenty CRM is the system-of-record for the sales pipeline + relationship layer — prospects, leads, opportunities, deals, sales notes.
Lifecycle: A prospect lives in Twenty as a Company + Opportunity. When Opportunity = Won → onboard to ITFlow. No automated sync; manual discipline for now.
One-line test: "Support or billing for an existing client?" → ITFlow. "A prospect, a deal, or the relationship?" → Twenty.
References: docs/data-ownership.md (standalone boundary reference), docs/engineer-hub.md (rule 7), docs/engineer-training-twenty-crm.md, docs/engineer-training-tacticalrmm-itflow.md.
Context: Discovered during NinjaOne → TacticalRMM client/site migration (Phase 1, 18 Jun 2026). Applicable to any TRMM integration or future white-label service built on this API.
Quirk 1 — No /api/v3 prefix. The TRMM REST API is rooted directly at the api. subdomain. All routes beginning /api/v3/* return 404. The correct base is https://api.<domain> (e.g. https://api.xzopiasecure.com/clients/, not /api/v3/clients/). The dashboard lives at rmm.<domain>; the API host is separate.
Quirk 2 — Sites are not a top-level resource. There is no GET /sites/ list endpoint. Sites are embedded in client detail: GET /clients/{id}/. The first site is created as part of the client create call (see Quirk 3). Additional sites go to POST /clients/sites/.
Quirk 3 — Client + first site created in one POST. POST /clients/ requires both client and site in a single nested payload:
{
"client": { "name": "...", "server_policy": null, "workstation_policy": null, "block_policy_inheritance": false },
"site": { "name": "Default Site" }
}
Subsequent sites for the same client: POST /clients/sites/ with {"site": {"name": "...", "client": <id>}}.
Quirk 4 — Auth header. Key is passed as X-API-KEY: <key> (not Bearer). The key is generated in the TRMM dashboard at Settings → API Keys.
Methodology rule — prove-out first for any bulk-create migration. Before looping 60+ creates: (1) fix the base URL and payload shape, (2) create ONE record, (3) read it back via GET /clients/{id}/ and confirm the response matches expectations, (4) confirm idempotency (re-running skips the record rather than duplicating or erroring), (5) only then release the full loop. This catches wrong base URLs and guessed payload shapes on one cheap, reversible record.
References: docs/migration/dry-run-report.md, docs/migration/phase1-clients-sites-report.md, src/commands/migrate-ninjaone-trmm.ts.
Context: FusionPBX gateway to VoiceHost (secure.st.sipconvergence.co.uk) returned 408 Request Timeout on re-registers after the initial 200 OK. Previously misdiagnosed as FreeSWITCH reusing auth nonces instead of requesting a fresh 401 challenge.
Root cause (confirmed 18 Jun via VoiceHost ticket #162431): VoiceHost's three cluster nodes each maintain their own per-source-IP block list. The accumulated failed registration attempts from debugging (multiple re-register failures while pinning to a single node) caused the node to silently block our source IP. A blocked node returns nothing — no 401, no 5xx — which FreeSWITCH surfaces as 408 Request Timeout. This is easy to misread as an auth/nonce incompatibility when it is actually a firewall block.
Diagnostic signature: If re-registers fail with 408 and a SIP trace shows no response at all from the remote node (not even a 401 challenge), suspect a per-node IP block before diagnosing the SIP stack.
Fix path: Get VoiceHost to clear the source-IP block for 77.68.112.222 on the pinned node (185.91.41.30). Enable SRTP. Re-test: watch a live sofia SIP trace (sofia global siptrace on in the FreeSWITCH console) for the full 401 → re-REGISTER with auth → 200 OK cycle. The reused-nonce → fresh-401 → reauth cycle is standard SIP behaviour and recovers automatically once the block is cleared.
Rule: Before debugging SIP auth or FreeSWITCH nonce handling, confirm the remote node is actually responding. Silence (→ 408) means blocked or unreachable, not auth failure.
References: FusionPBX build docs/server5-fusionpbx-build.md; VoiceHost ticket #162431 (b4b); LESSONS_LEARNED.md #78 (round-robin nonce misdiagnosis).
Context: NinjaOne → TacticalRMM migration (18 Jun 2026). Investigated whether the NinjaOne MCP (client OAuth2 credentials) could export policy configurations and script bodies programmatically.
What the API gives you: Device inventory, org/location metadata, policy names (get_policy returns name/description/nodeClass only), device activities log, alert conditions. Policy assignment per device via policyId / rolePolicyId.
What the API does NOT give you: Policy configuration bodies (schedules, alert thresholds, automation settings), script bodies (list_automations is blocked under client credentials), custom field values (returned empty).
Workaround — scripts-in-use via activities log: The device activities log (get_device_activities) records the name of every automation that fires, as sourceName on activityType = ACTION entries (format: "Run <AutomationName>"). This lets you build a targeted export checklist of scripts that have actually run on the fleet. Crawl all policy buckets (2–3 representative online devices per bucket, paginate backwards with after=<lastActivityId>). See docs/migration/ninjaone-scripts-in-use.md.
Critical caveat: The activities crawl only sees scheduled scripts (those that fire via automation policy). On-demand / manual scripts (e.g. Chocolatey onboarding, O365 offline install, version upgrade checks, one-off fixes) never appear in the activities log. Before cancelling a NinjaOne subscription, also pull the full Administration → Library → Automation (Scripting) index from the NinjaOne UI — that is the authoritative catalogue of all scripts regardless of schedule.
Rule for any NinjaOne tenant exit: (1) crawl activities log for scheduled-script names; (2) pull UI Library index for on-demand scripts; (3) map policy assignments via device policyId/rolePolicyId; (4) manually export script bodies from the UI (bodies are UI-only, only copy that exists).
NinjaOne API quirk to avoid: calling get_device_activities with activity_type=SCRIPT throws InvalidFilterException (HTTP 500). Always call with no activity_type filter; filter client-side on activityType ∈ {"ACTION", "SCRIPT"}.
References: docs/migration/ninjaone-scripts-in-use.md (crawl results, commit 2e95e83); docs/migration/ninjaone-policy-extraction.md.
Context: Researched 18 Jun 2026 during TacticalRMM policy design for the NinjaOne migration. These constraints shape how automation policies must be built.
Constraint 1 — No patch catch-up. If a managed device is offline at the scheduled patch install time, TRMM does NOT automatically retry on next online. Patches are simply missed until the next scheduled window. Mitigation (TRMM-native): add a scheduled CHECK that evaluates patch state (days since last update / pending critical count); on FAILURE, the check triggers a remediation TASK (PSWindowsUpdate script) that runs whenever the device is next online. This compensates for both the no-catch-up gap and occasional-off machines (e.g. N M Williams desktops off nights/weekends).
Constraint 2 — No native Wake-on-LAN in patch policies. WoL is not built into TRMM patch scheduling. If a device must be woken for patching, a script + an always-on same-subnet device is required.
Constraint 3 — Only ONE automation policy per level (client / site / agent). Do not attempt to stack multiple policies on the same object. Use the inheritance hierarchy (global default → client/site/agent override) and the block_policy_inheritance + Enforced flags. For mixed-device clients (e.g. Ziggi Cig: tills + back-office desktops), assign the override at SITE or AGENT level, not client level.
Constraint 4 — Decouple reboots from patch installs. Set "Reboot After Installation = Never" on patch policies and manage reboots via a separate scheduled task (e.g. Sunday 01:00–09:00 for servers; overnight for desktops). This prevents POS terminals and servers from rebooting mid-business.
Four-tier patch governance agreed 18 Jun: (1) Routine patches → per-client maintenance window; (2) Critical/security patches → fast lane (auto-approve Critical, test ring, next-online remediation); (3) Missed-patch auto-remediation → scheduled CHECK triggers PSWindowsUpdate task on next-online; (4) Zero-day manual override → pre-built library script, ring skipped, install-only vs install+reboot choice. Visibility: patch-compliance dashboard flag.
References: docs/tacticalrmm-policy-design.md.
Decision (18 Jun 2026): For TacticalRMM-delivered software installation and 3rd-party app patching, the engine is winget (Windows Package Manager). The Chocolatey community repository is NOT used for RMM-delivered installs.
Why not Chocolatey community repo:
Why winget:
Winget gotchas to handle in scripts:
winget upgrade --all WITHOUT --include-unknown — this skips apps whose version is opaque to winget (version = "Unknown") rather than endlessly re-installing them.winget pin) or exclude from the auto-sweep; update on a targeted cadence instead.Adobe.Acrobat.Reader.64-bit). On onboarding, install/reinstall Adobe Reader via winget to "adopt" the existing install into the upgrade lane — otherwise it sits permanently outside the managed upgrade path.packages/baseline.psd1 manifest (winget IDs) as single source of truth feeding both onboarding-install and the 3rd-party upgrade sweep. Keep optional per-device apps in packages/optional/ — install at agent level, not baseline.References: docs/scripts-library-standard.md; scripts/packages/baseline.psd1 (to be created).
actions array, not top-level script fieldDate: 22 June 2026
When creating TRMM automation tasks via POST /tasks/, the script to run must be provided in the actions array, not as a top-level script field:
{
"policy": 5,
"name": "Daily Maintenance",
"task_type": "scheduled",
"enabled": true,
"daily_interval": 1,
"run_time_date": "2026-06-23T01:00:00Z",
"timeout": 600,
"actions": [{"type": "script", "script": 132, "timeout": 600, "script_args": []}]
}
Passing "script": 132 at the top level is silently ignored — the task is created but actions is empty.
run_time_bit_weekdays encoding (Wednesday = 8): Based on the Windows Task Scheduler bitmask (Sun=1, Mon=2, Tue=4, Wed=8, Thu=16, Fri=32, Sat=64). Verify in the TRMM UI after task creation — the UI shows the scheduled day clearly.
run_asap_after_missed: true ensures offline agents run the task when they next come online, compensating for the no-catch-up limitation (Tier 3 missed-patch auto-remediation concept).
/clients/ "regression" was a misdiagnosis — payload-shape error, endpoint healthyDate: 22 June 2026 (corrected same day after reading django_debug.log and clients/views.py on S1)
The HTTP 500 initially recorded as a platform regression was caused by our ad-hoc calls sending the wrong payload shape. The endpoint is healthy and unchanged.
Root cause (confirmed from /rmm/api/tacticalrmm/private/log/django_debug.log):
KeyError: 'client' at clients/views.py line 75 (POST) and line 144 (PUT). The handlers require client fields wrapped under a "client" key — they do request.data['client'], so a flat top-level payload raises KeyError → Django returns 500. No regression, no server change: rmm.service had been running continuously since 1 Jun 2026 with no restart or update.
Correct payload contract:
POST /clients/ :
{ "client": { "name": "...", <client fields> },
"site": { "name": "Default Site" },
"custom_fields": [] // optional
}
PUT /clients/{id}/ :
{ "client": { "name": "...", <client fields incl. policy assignment> },
"custom_fields": [] // optional
}
The Phase 1 migration (commit 2306b3f) used this nested shape and succeeded. The 22 Jun ad-hoc calls dropped the wrapper — that's the only difference.
Rule: Before assuming a platform regression, check uptime / systemctl status rmm first. A service running continuously since before the problem date cannot have regressed. Then read the Django traceback — KeyError on a known field is a payload shape error, not a server bug.
Date: 22 June 2026
TRMM 1.4.1 is running on Server 1. TRMM 1.5.1 is available. Do not update during Phase 2 pilot or fleet migration. Defer until the pilot ring is confirmed healthy and at least one full client cutover has completed.
Why the risk is real for minor bumps (not just majors):
update.sh applies them immediately; there is no dry-run or rollback path short of a server snapshot.Procedure when applying (post-pilot):
update.sh in a maintenance windowReference: docs/trmm-policy-build-runbook.md §TRMM version and update policy.
Date: 22 June 2026
The NinjaOne get_device_activities API endpoint returns events from the activity log — script runs that were triggered on specific devices and recorded as events. It does NOT surface:
Specific errors this caused in our migration planning:
Rule: The activities-log crawl is useful for discovering WHICH scripts ran, but never trust it for schedule times or completeness. For any policy being reproduced in TRMM, manually verify in the NinjaOne Automation → Scheduled Automations tab before writing down the schedule. The tab shows Run Once automations, weekly/daily schedules, and the exact local-time value — none of which are derivable from the API.
Date: 22 June 2026
If you write a PowerShell function named Clear-RecycleBin, calling Clear-RecycleBin -Force inside that function calls the function recursively, not the built-in PS cmdlet. The function-scope lookup takes precedence over cmdlet discovery.
Workaround used in our library: The Clear-RecycleBin atom uses Get-PSDrive to walk $Recycle.Bin directly, avoiding the PS cmdlet entirely. This is also more reliable than the PS cmdlet when running as LocalSystem (which has no user profile).
General rule: If an atom function shares a name with an existing PS cmdlet, either rename the atom or implement the function body without calling the same-named cmdlet. Module-qualified calls (Microsoft.PowerShell.Management\Clear-RecycleBin) work but are fragile across PS versions.
PUT /clients/sites/{id}/ requires {"site": {...}} wrapper — same rule as clientsDate: 23 June 2026
PUT /clients/sites/{id}/ returns HTTP 500 if the payload is sent flat. The Django handler does request.data["site"] on the first line — a flat payload raises KeyError: 'site' → 500.
Correct shape:
PUT /clients/sites/66/
{ "site": { "workstation_policy": 12, "server_policy": 13 } }
Same pattern as PUT /clients/{id}/ which requires {"client": {...}} (see lesson #92). The pattern is consistent across TRMM's client-family endpoints: the handler always unwraps request.data["<resource_type>"] before the serializer.
Rule: When writing TRMM PUT calls against any clients/ or clients/sites/ endpoint, always wrap the payload under the resource-type key.
Date: 25 June 2026 (confirmed on pilot VM DESKTOP-CC2U7F7 / NinjaOne device 564)
What happened: TRMM Phase 2 pilot install on a test VM appeared to succeed in NinjaOne (script exit 0) but no tacticalrmm service was created and the agent never appeared in TRMM. The agent binary reached C:\Program Files\TacticalAgent but never registered (0-byte agent.log). Root cause: SentinelOne Threat History showed tacticalagent-v2.10.0-windows-amd64.exe flagged "Malicious file executed → Risk Mitigated → Quarantined." S1 quarantines the installer the instant it executes ("Access is denied"). The outer PS1 wrapper exits 0 — S1's quarantine is silent to NinjaOne.
Why it matters: RMM/remote-control agents look like attacker tooling to EDRs (classified as riskware/hacktool). Every Xzopia fleet endpoint runs SentinelOne → without exclusions, the rollout fails silently on every machine. The bundled MeshCentral agent is also quarantined if C:\Program Files\Mesh Agent\* is not in the exclusion set — this path gap caused the subsequent mesh/take-control failures even after S1 exclusions were applied.
THE FIX — mandatory Step-0 pre-rollout gate: In the SentinelOne management console, create exclusions scoped fleet-wide, of a type that permits execution (interoperability / path-exclusion, not scan-only):
tacticalagent-*.exe (+ SHA1 hash from the quarantine entry)C:\Program Files\TacticalAgent\*C:\Program Files\Mesh Agent\* ← do NOT omit; absence causes mesh/VNC/take-control failureC:\ProgramData\TacticalRMM\*Follow TRMM's published SentinelOne exclusion guidance for the authoritative path/hash list and recommended mode.
Knock-on — TeamViewer Host is in the same class: the branded Host installer and process will likely also be quarantined by S1. Add TeamViewer_Host_Setup*.exe + C:\Program Files\TeamViewer\* to the same exclusion set before the TeamViewer fleet deploy.
Rule: Any RMM agent, remote-control agent, or security tooling deploy on SentinelOne-managed endpoints must be preceded by execution-permitting S1 exclusions scoped to the full fleet. Step-0, not an afterthought.
systemctl reload nats requiredDate: 25 June 2026 (confirmed on pilot VM DESKTOP-CC2U7F7 / S1 213.171.194.170)
What happened: After a new TRMM agent registered on Server 1, the agent entered an ~10s crash-loop. Agent log: "Agent service started" then level=error msg="EOF" repeating. NATS log showed authentication error for the new agent's user from its endpoint IP, looping every ~10 seconds. The on-disk nats-rmm.conf already contained the correct credentials — TRMM's registration had written them correctly.
Root cause: NATS loads its configuration on startup and keeps the user list in memory. TRMM writes new agent credentials to nats-rmm.conf at registration time but does NOT signal NATS to reload. NATS keeps rejecting the new agent until it receives SIGHUP.
Symptoms: EOF crash-loop; agent stuck at v0.1.0 placeholder; checks return HTTP 400; "Recover Connection" shows "bad agent."
Fix: sudo systemctl reload nats (SIGHUP — non-destructive, no existing connections dropped). Post-reload: authentication errors stop within seconds; agent version updates to 2.10.0 within ~7 seconds and communicates normally.
Evidence: NATS log showed the ~10s authentication error loop for the new agent's endpoint IP. Loop stopped immediately after reload; agent communicated normally thereafter.
Permanent fix (26 June 2026): A systemd path unit now watches nats-rmm.conf for changes and triggers systemctl reload nats automatically. Two unit files deployed to S1:
/etc/systemd/system/nats-reload.path — PathModified=/rmm/api/tacticalrmm/nats-rmm.conf, enabled and active/etc/systemd/system/nats-reload.service — Type=oneshot, ExecStart=/bin/systemctl reload natsThe path unit uses IN_CLOSE_WRITE (file closed after write). Python's with open(...) as f: json.dump(...) triggers this correctly. systemctl reload nats fires ~0.5 s before TRMM's own subprocess.run signal attempt (which fails silently from within a uwsgi worker — see root cause below). Manual systemctl reload nats is no longer needed after registration.
Root cause of the subprocess failure: nats-server -signal reload exits 0 silently but does NOT send SIGHUP when called via subprocess.run inside a running uwsgi worker process (master=true, enable-threads=true). The exact mechanism is unknown — no sandbox restrictions on rmm.service, same user, same binary. The call works correctly from SSH, Django management command, and Python subprocesses outside uwsgi. The path unit bypasses this entirely by operating at the OS level.
Note: This is distinct from the SentinelOne installer-quarantine issue (lesson #97) — that is a Windows-side problem. This is a server-side NATS problem affecting agents that have already installed and registered.
Date: 25 June 2026 (confirmed on pilot VM DESKTOP-CC2U7F7 / S1 213.171.194.170)
What happened: The TRMM Take Control button showed 🚫. All mesh API XHRs returned 200 throughout. MeshCentral's own Desktop tab worked perfectly when accessed directly (unframed, top-level). Browser console: "Refused to display 'https://mesh.xzopiasecure.com/' in a frame because it set 'X-Frame-Options'." Agent health, token validity, gotonode URL, and mesh permissions were all correct the entire time.
Root cause: The mesh nginx vhost included snippets/xzopia-security.conf (our shared hardening snippet), which emits add_header X-Frame-Options "SAMEORIGIN" always;. TRMM opens MeshCentral in a cross-origin iframe (rmm.xzopiasecure.com → mesh.xzopiasecure.com). SAMEORIGIN blocks all cross-origin framing — there is no per-origin allowlist for X-Frame-Options. MeshCentral's own allowFraming: true in its config.json was being overridden by nginx.
The asymmetry fingerprint: MeshCentral Desktop tab works (top-level, unframed) while TRMM Take Control blocks (cross-origin iframe). If the issue were agent/relay/token, both paths would fail. If only the framed path fails while the unframed path works, the cause is a framing header.
Fix (mesh vhost only):
include snippets/xzopia-security.conf;X-Frame-Options with: add_header Content-Security-Policy "frame-ancestors 'self' https://rmm.xzopiasecure.com" always;X-Frame-Options SAMEORIGIN untouchedBackup at /etc/nginx/sites-available/meshcentral.conf.bak-20260625. Verified via curl -skI: X-Frame-Options absent, Content-Security-Policy: frame-ancestors 'self' https://rmm.xzopiasecure.com present; Take Control desktop paints.
Investigated-not-causal — mesh-perms sync: sync_mesh_with_trmm was run mid-debug as a theory. It was NOT the cause. Simon was already synced to the MeshCentral node throughout (web-RDP worked via a separate code path). Do not treat the sync as a fix.
Pentest note: Lynis and future pentest scans will flag mesh.xzopiasecure.com for a missing X-Frame-Options header. This is intentional — the frame-ancestors CSP is the correct mechanism for cross-origin framing allowlists. Whitelist this finding.
Rule: Security-header snippets applied globally will break TRMM Take Control. Per-vhost headers are mandatory for any stack where TRMM frames MeshCentral cross-origin. See docs/security/header-policy.md.
Date: 26 June 2026 (root-caused on S1 213.171.194.170; affects TRMM 1.4.0)
What happened: Celery Beat and the Celery worker had both been unable to connect to Redis for 25 days (since June 1 23:18). Beat accumulated 17.5M error lines and a 17 MB log. All periodic Celery tasks were not running: no agent outage detection, no sync_mesh_perms_task, no trmm-scheduler (which dispatches ONBOARDING tasks), no maintenance tasks. Worker logs were nearly empty — workers were not receiving any task dispatches.
Root cause: A Lynis DBS-1884 hardening session on June 1 23:17–23:18 modified redis.conf (to add/remove requirepass) and local_settings.py (to add the Lynis comment). The exact sequence created a mismatch between what the running Beat and worker processes had in their connection pools and what Redis then expected. The connection pools became permanently broken — each reconnect attempt produced AuthenticationError: AUTH <password> called without any password configured for the default user. Both services were never restarted after the config change, so the broken pool persisted indefinitely.
Why it lasted: Celery/Kombu catches the auth error per task dispatch, logs it, and tries again next tick. There is no circuit-breaker or self-healing restart. With tasks every 80–150 seconds, the log fills at ~8 lines/second.
The config mismatch (Lynis hardening left-incomplete):
redis.conf has #requirepass 7IvMBeW6FzTBJ5y4Zh8Urg6kDPSpkGkKzKGaWBJr (commented out)local_settings.py has # Redis password — Lynis DBS-1884 hardening comment above REDIS_HOST = "127.0.0.1" (no password in URL)redis.Redis.from_url("redis://127.0.0.1") connection works correctly — the issue is specific to the long-running process poolIf Lynis DBS-1884 is applied properly in future: both redis.conf (requirepass <password>) AND local_settings.py (REDIS_HOST = ":<password>@127.0.0.1") must be updated together, and Beat + worker must be restarted in the same step.
Fix: Rotate logs, restart both services:
sudo truncate -s 0 /var/log/celery/beat.log
sudo truncate -s 0 /var/log/celery/w1.log
sudo systemctl restart celerybeat
sudo systemctl restart celery
# Verify after ~30s:
sudo wc -l /var/log/celery/beat.log /var/log/celery/w1.log
Post-restart: both logs remained at 0 lines for 2+ minutes. Beat dispatches tasks. Workers receive and execute them. CoreSettingsNotFound errors in the worker log on first start are transient — cache-db-fields-task populates the cache within 3 minutes and subsequent task runs succeed.
Rule: Any time redis.conf or local_settings.py Redis settings are changed, restart celerybeat and celery immediately in the same change window. Do not rely on the running processes picking up changes — they do not.
Date: 27 June 2026 (root-caused while loading tv-deploy.ps1 into TRMM)
What happened: tv-deploy.ps1 was correct on disk but failed to parse the moment it was pasted into the TRMM script editor. Output was a cascade of "missing terminator in string" and "missing closing '}' in statement block" errors. The script had not been edited — TRMM had mangled it.
Root cause: The script contained non-ASCII characters — em-dashes (—, U+2014) inside throw/Write-Output strings, box-drawing comment dividers (──, U+2500), and one arrow (→, U+2192). TRMM's web script editor does not preserve these multi-byte UTF-8 bytes; it replaced each with ?. A ? inside a single-quoted string is harmless, but the corruption desynchronised the parser's view of where strings and blocks opened/closed, producing terminator/brace errors far from the real location.
Fix: Convert the script to pure ASCII before loading — em-dashes → -, box dividers → # ---, smart quotes → straight quotes, arrows → ->. Byte count dropped 10,439 → 9,594 with zero logic change.
Verification (run before every endpoint-script deploy):
grep -P '[^\x00-\x7F]' <script> # must return NOTHING
file <script> # should report "ASCII text"
Rule: Any script pasted into the TRMM editor (PowerShell or bash, all of scripts/windows/**) must be pure ASCII. This is now a standing rule in CLAUDE.md. Repo markdown is exempt — it renders to the wiki, not a TRMM textbox.
exit exits the scope, not the process — use throw + a single outer catch to halt reliablyDate: 27 June 2026 (root-caused from the pilot trace on DESKTOP-CC2U7F7 via tv-deploy.ps1; fix CONFIRMED by a clean pilot run same day)
Status: CONFIRMED. Root cause confirmed from the pilot trace; the throw + single-outer-catch fix is now proven by a clean TRMM run on DESKTOP-CC2U7F7 — the restructured (and ASCII-clean, see #101) script executed the full sequence end-to-end: uninstall full client → download (Size 35843272 / SHA1 C08A0466… / Integrity OK) → install /S /norestart → exit 0 → "OK: TeamViewer service is running." The throw/outer-catch structure ran through the success path cleanly; no spurious skipped-stage behaviour.
What happened: The first tv-deploy.ps1 pilot run printed Authorizing B2 download... then jumped straight to Verifying integrity... — the entire download block appeared skipped, and no installer file was written. The auth had actually failed (a trailing-whitespace 401 on the read key), and each try/catch block aborted with exit 1, yet execution carried on into later stages.
Root cause: TRMM runs endpoint scripts inside an agent-hosted PowerShell runspace, not a standalone powershell.exe process. In a runspace, exit terminates the current scope, not the process — so exit 1 inside a catch did not halt the run. With $auth never assigned after the failed auth, the subsequent stage dereferenced $auth.downloadUrl under Set-StrictMode -Version Latest, throwing a "property on null" terminating error that the next catch swallowed with another non-halting exit 1. The visible trace looked like blocks were skipped because the Write-Output lines and the real exception text landed on different streams that TRMM interleaves out of order.
Fix: Restructure to a single fail-fast pipeline — every stage either succeeds or throws; one outer try wraps the whole sequence; the single outer catch reports the real message and is the only place exit 1 is used. throw always unwinds the runspace, so a failed stage can never reach a later stage regardless of exit semantics. Also added: .Trim() on the key params (the actual 401 cause), and an explicit check that the auth response carries downloadUrl/authorizationToken before use. Verified: the clean pilot run on DESKTOP-CC2U7F7 completed the full success path end-to-end (download → integrity OK → install → service running → exit 0), confirming the restructured pipeline executes correctly under the TRMM runspace.
Rule: In any TRMM-deployed PowerShell script, do not rely on exit N to stop execution mid-script. Use throw for inner failures and a single outer try/catch that exits once. Treat StrictMode property access on a possibly-unset variable as a hard error path — validate before dereferencing.
ClientID is a local cache, not the source; captured IDs are stable across redeploysDate: 27 June 2026 (CONFIRMED on pilot DESKTOP-CC2U7F7)
What happened: A full TeamViewer uninstall (which removes the ClientID registry key) followed by a fresh branded-Host reinstall on the same hardware produced the same TeamViewer ID — 707257581 — after the new Host registered. The earlier working assumption (handoff, 26 Jun) was that the uninstaller wiping ClientID meant the fresh Host would be assigned a new ID.
Root cause: The registry ClientID (HKLM\SOFTWARE\TeamViewer\ClientID, 64-bit; WOW6432Node on 32-bit) is a local cache of a server-assigned, machine-bound ID — not the source of truth. TeamViewer derives the ID from a hardware fingerprint server-side, so on re-registration from the same machine the server returns the same ID and repopulates the same registry value. Wiping the local key does not orphan or rotate the ID.
Implications:
teamviewer10://control?device={{agent.TeamViewerID}} URL Actions remain valid.tv-capture only needs to populate the TRMM custom field once per machine; it does not need to re-run after every redeploy (a weekly sweep is still fine as a backstop / for new machines).ClientID is absent — between uninstall and the fresh Host's first registration — so tv-capture must remain a separate, later task, not chained immediately after tv-deploy. The value is stable; the timing of its reappearance is the only variable.Rule: Treat a device's TeamViewer ID as a durable, hardware-bound identifier. Capture once it appears post-registration; do not assume a redeploy invalidates it.
Date: 27 June 2026 (CONFIRMED — all three paths investigated and ruled out on TV 15.79.4, Premium licence)
What happened: The goal was a fully-scripted TeamViewer Host deploy where the device is also reachable unattended (no end-user present) via a set password, so a tech could click a connect button and get straight in. The branded-Host install scripts cleanly (lessons #101–#103), but every attempt to set the unattended access password by script failed. All three documented methods were tried and each is blocked on our tier.
The three paths, all closed on Premium + v15.x SRP:
--grant-easy-access / assignment API) — Corporate/Tensor only. The manual assignment ran but was auth-rejected on connect; TeamViewer's own config tooltip confirms automatic easy-access (no-confirmation connection) requires the higher-tier flow. Not available on Premium.SETTINGSFILE=<.tvopt> (the documented clean password-deploy) — the MSI installer is Corporate/Tensor only; on Premium we only have the EXE, which does not accept SETTINGSFILE. The older IMPORTREGFILE argument has been deprecated since 15.4. No path on the EXE.PermanentPassword on one machine, replay on others) — dead on v15.x. The password is stored as PermanentPassword (REG_BINARY) bound to SRPPasswordMachineIdentifier (per-machine Secure Remote Password). The hash is not portable between machines — the modern SRP hardening specifically killed cross-machine replay. Replaying it would be fragile, undocumented registry surgery that breaks on every TV update. Rejected.Decision / landed model (do NOT re-litigate):
tv-deploy.ps1) → set the personal password MANUALLY, once, during onboarding (a tech is already on the box at onboarding; ~30 sec; only new devices — the existing fleet already has passwords).teamviewer10://control?device={{agent.TeamViewerID}} opens the session and the tech enters the onboarding password.Rule: Do not attempt to script the TeamViewer unattended-access password on a Premium licence — it is a licence/tier wall, not a technical problem to brute-force. The only clean fully-scripted path is a Corporate/Tensor upgrade (unlocks Easy Access + MSI SETTINGSFILE), which is a business cost/benefit decision. For now: scripted Host install + manual onboarding password + MeshCentral for unattended.
HKEY_USERS\<SID> to inspect a specific userDate: 1 July 2026 (CONFIRMED — root cause of 8-month malware survival on the O'Boyle Accounting machine)
What happened: Malware persistence written to the interactive user's HKCU\...\Run survived for ~8 months despite SYSTEM-context scanning. A process running as SYSTEM (the TRMM/SentinelOne agent, a scheduled task, or any -RunAsSystem script) resolves HKCU to SYSTEM's profile hive (S-1-5-18), not the logged-on user's. So a SYSTEM-context scan that reads HKCU:\Software\Microsoft\Windows\CurrentVersion\Run never saw the user-hive Run keys where the persistence actually lived.
Root cause: HKEY_CURRENT_USER is a per-process view that maps to whichever account the process token belongs to. Under SYSTEM that is HKEY_USERS\S-1-5-18, which has no relationship to the human user's hive.
Rule: When running as SYSTEM and you need a specific user's HKCU, mount their hive explicitly:
New-PSDrive -Name HKU -PSProvider Registry -Root HKEY_USERS | Out-Null
# then read HKU:\<SID>\Software\... (enumerate loaded SIDs under HKEY_USERS;
# for a logged-off user, reg load HKU\Temp <path>\NTUSER.DAT first, then unload)
Never assume HKCU == the interactive user when the process token is SYSTEM. Any endpoint scan for user-scoped persistence must iterate the real user SIDs under HKEY_USERS.
Date: 1 July 2026 (rule violations observed and corrected)
What happened: Chat created wiki pages id=61 and id=63 directly and edited the cc-prompts index (id=43) directly. All three are violations of the CC↔chat documentation workflow: chat may write only the session-handoff page (id=15). Repo-worthy items (lessons, status changes, design docs) and new cc-prompt pages must be staged in id=15 under a "TO COMMIT (Claude Code)" block; Claude Code then creates the pages and commits the repo.
Root cause: The original pattern left "who creates a new cc-prompt page" ambiguous, so chat created them directly. The wiki then drifts ahead of the repo and there are two writers to the source of truth.
Rule: One writer per source of truth. Chat → id=15 only. Claude Code is the sole creator of every non-handoff wiki page (cc-prompts, cc-reports) and the sole repo committer. New prompt pages are staged in id=15 first, then CC creates them. id=15 is the only wiki doc permitted to be temporarily ahead of the repo; CC reconciles it into the repo and back out to the wiki.
Date: 1 July 2026 (CONFIRMED — O'Boyle DLL analysis, Stack360Hui634094.dll)
What happened: The recovered DLL carried Firefox/Gecko NSS crypto-library strings — the library browser stealers use to decrypt Firefox logins.json / key4.db. It was ineffective on this machine because the user runs Edge only (no Firefox profile to attack). Chrome/Edge credentials are DPAPI-protected, not NSS, so this module could never target them. The kit bundled all browser modules regardless of what was installed. Invocation was rundll32 + an EntryPoint export (classic malware DLL invocation), and the Run-key entry was disguised with a fake vendor name ("Palo Alto Network Sensor"). SentinelOne killed the PowerShell loaders on every execution (~126×), but the rundll32-hosted DLL ran as a separate process and appeared to evade flagging.
Rule: NSS strings in a suspicious binary = a Firefox-targeting credential stealer. Scope its impact to the browsers actually installed (no Firefox ⇒ this module is inert) — but treat the host as compromised regardless. Flag rundll32 + EntryPoint exports and fake-vendor Run-key names as evasion signals; a component that "did nothing" because of a missing target is not evidence the box was safe.
Date: 2 July 2026 (Denis Humphrey Solicitors)
What happened: A Yealink handset connected via a WiFi USB dongle (WF40/WF50-type) intermittently drops its WiFi association and loses PBX registration — the phone still shows "connected" but calls fail; a power-cycle restores it temporarily. The site WiFi is otherwise strong and the PBX is client-owned and untouched.
Root cause: The dongle fails to re-establish association after an AP channel/band change or reboot. Diagnostic tell: if wired handsets stay registered while only dongle-equipped handsets drop, the dongle is the cause — not the network, not the PBX.
Rule: Durable fix = run a wired data drop to the room, then disable WiFi / remove the dongle on the handset so it cannot fall back to it. Verify on copper: the phone pulls DHCP on the correct voice VLAN, re-registers, and the switch port is PoE if the handset is powered over the cable. Isolate to dongle handsets before blaming the network or PBX.
Date: 2 July 2026 (collision observed and repaired)
What happened: During a long CC task, chat updated id=15 live (marking a status "KEY GATE CLEARED") while CC was finishing the same task and then went to "reconcile" id=15 itself. Both wrote the same page in the same window — a direct violation of "one writer at a time". Two failures compounded: (1) the two writers raced, and (2) CC's reconcile used a greedy substring match ("KEY GATE CLEARED (Simon") that hit three lines, overwriting the agenda line and a table row with duplicated banner text and breaking the table. CC had to detect and repair it.
Root cause: id=15 had two writers. The "CC reconciles id=15" habit drifted in from task phrasing ("reconcile id=15") being read as "edit the wiki id=15", when the canonical rule was already "CC writes report pages ONLY — never id=15". Nothing enforced single-writer, so a timing overlap became real damage — amplified by a non-surgical edit.
Rule:
=== id=15 UPDATES TO APPLY === block (in its cc-report + final chat message) with exact edits; chat applies them. "Reconcile id=15" = read it into the repo (if needed) + emit that block, never edit the wiki page.Refinements (2–8 July, from live use):
=== id=15 UPDATES TO APPLY === block in both its cc-report page and its final chat message, identical. (2 Jul: report id=70 omitted it; the chat copy saved the hand-off — the dual-emit is the reliability backstop, not redundancy.)ListTenants finding, and separately mislabelled the CIPP Function App as "the abandoned self-hosted deployment" — both because CC acted on a stale/partial in-context copy of id=15. A skim is not enough; re-read the current page fully before reasoning about state.)Date: 2 July 2026 (S4 hardening, cc-prompt id=69 — all 11 MCP connectors preserved through config + reboot)
Three non-obvious traps hit while hardening Server 4 (SSH on 2222, 11 live MCP connectors). Each looks fine until you check the effective state.
fail2ban bans the wrong port when SSH is on a non-standard port. The [sshd] jail defaulted to port = ssh (resolves to 22). Detection still worked (it reads /var/log/auth.log regardless), so fail2ban-client status sshd showed bans — but the ban action was firewalling port 22, which nothing listens on, so brute-forcers on 2222 were never actually blocked. Fix: set port = 2222 in the [sshd] block of /etc/fail2ban/jail.local (never jail.conf). Verify the ban action targets the real port.
sshd_config.d drop-ins are first-match-wins, and cloud images ship an early one. Include /etc/ssh/sshd_config.d/*.conf sits near the top of sshd_config, and 50-cloud-init.conf sets PasswordAuthentication yes. Because sshd uses the first value obtained, a 60-/99- drop-in or the main-config line cannot override it. sshd -T (not the file contents) is the source of truth. Fix: put the override in a drop-in that sorts before 50- (e.g. 00-xzopia-hardening.conf) so it wins the first match. Always confirm with sshd -T | grep -E 'passwordauthentication|permitrootlogin' before reloading.
conf.all.* sysctls (e.g. log_martians) get reset after systemd-sysctl by systemd-networkd NIC init. sysctl --system applied net.ipv4.conf.all.log_martians=1 correctly, and it survived at apply-time — but after reboot it read 0, because the NIC is initialised (systemd-networkd) after systemd-sysctl runs, resetting some per-interface conf.all.* values to defaults. Global keys (e.g. net.core.bpf_jit_harden) are unaffected. Fix: a networkd-dispatcher reapply hook — /etc/networkd-dispatcher/routable.d/50-reapply-sysctl running sysctl --system when an interface becomes routable. Verify by forcing the value to 0, running the hook, and confirming it restores.
Meta-rule: on any hardening pass, verify the effective runtime state (sshd -T, sysctl -n, fail2ban-client status) after a reboot — not just that the config file contains the right line. And on a live MCP host, re-check every connector after each change and after the reboot; roll back to the snapshot if any is unreachable.
S1-EDR = SentinelOne, bare S1 = Server 1Date: 3 July 2026 (locked by Simon; topology-correction commit)
"S1" was overloaded: it meant both Server 1 (TacticalRMM + MeshCentral) and SentinelOne (the EDR product/console/agents). On a fleet where Server 1 and the EDR run on different infrastructure, that ambiguity is a real hazard (e.g. "close the 127 S1 incidents" vs "reboot S1").
Convention (all new writing): S1-EDR = SentinelOne. Bare S1 = Server 1 only. Older pages (cc-reports, BookStack, earlier lessons) may still use bare "S1" for SentinelOne — read those by context; do not retro-edit history, just disambiguate going forward.
Meta-rule: when one short token names two unrelated things, lock a distinguishing convention early and record it where every agent reads it (CLAUDE.md + this file) — abbreviations that are obvious to the author are a trap for a model or teammate resuming cold.
wiki_patch_page — surgical single-anchor Wiki.js edits (atomic-edit rule, mechanised)Date: 3 July 2026 (mcp-wikijs, S4 :3004; validated on scratch wiki page id=77)
Full-content wiki_update_page rewrites the whole page for every edit — a ~36KB round-trip for a 2KB change, plus full-page clobber risk (the exact failure mode LESSON #109's single-writer + unique-anchor rule guards against by hand). Added wiki_patch_page(id, old_str, new_str[, expected_hash]) to mcp-servers/wikijs/src/index.js: fetch content → require old_str occurs EXACTLY once (0 or >1 → reject, no write, return the match count) → literal single-occurrence replace → GraphQL update → re-fetch and verify the change landed. Optional expected_hash (SHA-256 of current content) is a concurrency guard. It mechanises the atomic-edit rule so it can't be forgotten.
Implementation notes:
split(old_str).join(new_str), not String.replace — a string new_str containing $&/$1 is interpreted as a replacement pattern by replace(); split/join treats it literally. (Since count is guaranteed 1, split/join replaces exactly the one occurrence.)403 Forbidden (not a GraphQL error). Space rapid automated calls; normal one-at-a-time use is fine. Same endpoint/limit applies to wiki_get_page/wiki_update_page.Rollout: id=15 (and cc-report) edits switch to wiki_patch_page; wiki_update_page retained for page creation/restructure only. A newly-added MCP tool is only visible to a connector after it reconnects (new chat) — the tool is live on S4 immediately, but existing sessions keep their old tool list until they reconnect.
restic prune is incompatible with an active B2 Object Lock — design for append-onlyDate: 3 July 2026 (commit c23d488, report id=74)
restic prune deletes locked pack/index objects, and restic 0.16 sends no governance-bypass header — so prune fundamentally cannot run against a bucket with an active B2 Object Lock, and a bypass-capable key does not fix it (restic never sends the bypass). restic forget without --prune only removes snapshot metadata older than the keep policy (≥ retention age) → safe under a short lock.
Design: gate the backup scripts on an OBJECT_LOCK flag = append-only (forget-only) under lock; reclaim space later via a planned privileged op (repo rotation / master-key bypass). Use governance (not compliance) mode with a short default retention (7d, aligned to keep-daily) — a mis-set lock is then self-healing (it lapses within a week). The lock (not the key's caps) enforces immutability: a compromised server key that still has deleteFiles cannot delete locked objects.
pgrep/ps a process while a secret is on its command lineDate: 3 July 2026
A pgrep -af probe of a running pro attach <token> surfaced the Ubuntu Pro token in plaintext in the CC transcript. Any secret on a process argv is world-readable via ps / pgrep / /proc/<pid>/cmdline for the whole life of the process. (A free-tier Pro token can't actually be rotated on demand — see #120 — so the fix here is don't leak it, not "rotate after".)
Rule: never inspect a live process cmdline while a secret is on argv — verify via exit code or tool status (pro status) instead. Prefer token-via-file/stdin over token-on-argv wherever the tool supports it; even a short-lived argv secret is exposed to any reader while the process runs.
Date: 3 July 2026
CC has no Vaultwarden-read tool and no bw CLI on the workstation, so it cannot pro attach straight from VW. Pattern: Simon file-drops the token to /root/.ua-token (mode 600) in his own SSH session; CC attaches from the file, then shred -us it. The token never enters chat or the repo. Free personal tier = 5 machines; one token attaches all boxes. (Do not pgrep/ps the running attach — #114.)
export prefixDate: 3 July 2026
/etc/xzopia-backup/credentials is sourced by the backup script, so its vars are written export VAR=... (the script needs them exported). A sed that matches only ^VAR= misses export VAR= and silently no-ops. Match the optional prefix and keep it:
sed 's|^\(export \)\?VAR=.*|export VAR="…"|'
(First attempt missed export HEALTHCHECK_UUID="".)
Date: 3 July 2026
Restricting an app-only Mail.Send to specific mailboxes: use pwsh 7 + EXO module 3.10.x (NOT Windows PowerShell 5.1). The module's "upgrade to V3.7" banner is stale — ignore it. Connect with -LoadCmdletHelp on 3.7+. Scope a mail-enabled security group, New-ApplicationAccessPolicy … -AccessRight RestrictAccess, then verify with Test-ApplicationAccessPolicy (Granted/Denied). ~1 hr propagation.
Date: 3 July 2026
Publisher-verifying an app registration needs the Partner Center PartnerGlobal ID (PGA) — a location ID returns 404. Prereqs: tenant associated in Partner Center, publisher domain verified, MFA + the right roles. Can fail and then succeed on retry (propagation). Confirmed on "Xzopia Automation" (PGA 1071553).
Date: 3 July 2026
id=15 said "build S2 backup (not built)"; wiki id=23 + repo docs + the live server all showed S2/S5 backup was built + test-restored weeks earlier. A blind rebuild would have re-init'd the repo and clobbered the DR-keystone's snapshots. The inverse also bit the same day: an "S4 unattended-upgrades has drifted" prior was wrong — logs proved every box was applying updates.
Rule: a "build/fix X" task must be reconciled against repo + existing wiki docs + live-server inventory before touching anything, especially on a live/keystone box. "Documented complete" ≠ "still live" — re-check a backup's last-snapshot date (a silently-stopped backup is worse than none). Verify both the assumed-good and the assumed-bad framing; the handoff can be stale in either direction.
Date: 6 July 2026 (Simon)
The free-personal Ubuntu Pro dashboard shows one fixed token; detaching and reattaching machines returns the same string. So a leaked free-tier token cannot be invalidated on demand — there is no rotate button. This corrects the earlier "rotate the exposed Pro token" action (3 Jul): on the free tier that action is impossible.
Response to a free-tier token leak:
pro detach the fleet in response to a leak — it drops Livepatch (live kernel-CVE protection) across S1/S2/S4 for zero security gain, since the same token re-attaches anyway.pgrep/ps the running attach).Paid Ubuntu Pro tiers differ — their tokens are rotatable, so this constraint is specific to the free-personal tier (5 machines).
resolve_threat silently no-ops on a threat with no analyst verdict — set the verdict FIRSTDate: 8 July 2026 (found closing out O'Boyle)
Bulk-resolving the 127 held S1-EDR incidents on the O'Boyle site, resolve_threat (POST /threats/incident, incidentStatus) returned affected: 0 with no error on any threat whose analystVerdict was still undefined. It looks like it worked (HTTP 200, no exception) but the incident status never changes — silent, easy to miss at scale. Calling set_analyst_verdict (POST /threats/analyst-verdict, e.g. true_positive/false_positive) first, then resolve_threat, makes the resolve take effect. SentinelOne treats a verdict as a precondition for an incident-status transition and reports the unmet precondition as a zero-affected success, not a failure.
Separately, a single resolve_threat call with 127 IDs only affected 80 on the first pass (the front 47 of the array failed) — an apparent ~80-item batch ceiling on the incident endpoint, independent of the verdict issue. Exact ceiling not pinned down.
Rule: for any bulk S1-EDR closure — always set_analyst_verdict before resolve_threat, and assert on affected == requested, never on the absence of an error (a zero-affected "success" is the failure mode here). Chunk bulk resolves to ≤80 IDs per call and loop until affected matches the batch. Bake both into any future bulk-closure runbook/tooling; consider having the MCP resolve_threat surface a warning (or auto-set a verdict) when it resolves a verdict-less threat. See the S1-EDR MCP (mcp-servers/sentinelone/src/index.js, set_analyst_verdict + resolve_threat).
get_agent_passphrase — the passphrase lives on a filtered collection endpoint, not a per-agent sub-resourceDate: 8 July 2026
The get_agent_passphrase tool called GET /agents/{id}/passphrase and always 404'd — that sub-resource does not exist in the SentinelOne v2.1 API. The passphrase is exposed via the filtered collection endpoint: GET /web/api/v2.1/agents/passphrases?ids={id} (comma-separated IDs), the same shape as get_agent (GET /agents?ids={id}). Corroborated by SentinelOne's own Get-SentinelOneAgentPassphrases cmdlet and Postman collection. Fixed in mcp-servers/sentinelone/src/index.js to build a query string (ids=) against /agents/passphrases instead of interpolating the ID into the path. (Console workaround while a fix ships: agent → Actions → Show Passphrase; a console-driven uninstall doesn't need it.)
Rule: SentinelOne v2.1 exposes per-entity secondary data (passphrases, and similar) through filtered collection endpoints keyed by an ids query param, not REST-style /{id}/sub-resource paths — mirror the existing get_agent/get_threat ?ids= pattern for any new per-agent lookup rather than assuming a nested path. A repo code fix to an S4-hosted MCP is not live until deployed to S4 (backup-first, then re-verify the connector) — committing the fix and shipping it are two steps.
Date: 8 July 2026
Removing a client from the RMM (NinjaOne / TacticalRMM) leaves the SentinelOne (S1-EDR) agent live: the machine keeps phoning home and consuming an EDR licence seat. BSA had 2 live S1-EDR agents after departure because RMM removal was treated as the whole offboarding. Managed platforms do not cascade — each (S1-EDR, RMM, PSA/ITFlow, Vaultwarden, M365/CIPP/GDAP, backups/DNS/VoIP) must be offboarded on its own and the result verified ("removed from the console" ≠ "confirmed gone").
Rule: follow docs/runbooks/client-offboarding.md for every departure, with the S1-EDR agent-removal step first (Uninstall live / Decommission offline → site empty → retire site). Run scripts/server4/s1-edr-orphaned-agent-audit.sh (read-only) after every offboarding and monthly to catch agents missed historically — it flags expired-site / stale (no recent lastActiveDate) / outdated agents. A legitimately powered-off spare will look stale, so review each flag before decommissioning.
Date: 8 July 2026
Reasoning from a stale or partial in-context copy of id=15 has produced wrong output repeatedly — and not only when writing, also when investigating. Three proof points:
ListTenants root cause back to "unconfirmed / candidate ExecGDAPTrace" because it did not re-incorporate what id=15 already recorded; had to be self-corrected.c28a312) had gone stale mid-session (it predated the drill correction) — re-committed the current version instead of re-shipping the stale one.docs/runbooks/disaster-recovery.md file, and flagged the exact wiki-sync-mirror fix — the same "verify the primary source" discipline showing up independently.Rule: at the start of every task — investigative/diagnostic included, not just before writing — fetch and read the whole current id=15, never a skim or an in-context copy from earlier in the session (another turn or another session may have changed it). More broadly, verify against the primary source — the repo, the live system, the real file — not a secondhand summary. id=15's own TO COMMIT list can be stale: an item may already be committed (e.g. via a reconcile commit) and simply not yet reconciled out, so treat "still listed as TO COMMIT" as a claim to check against the repo, not a fact. Related: #119 (verify ground truth both ways before build/fix).
Date: 8 July 2026
The DR runbook draft (chat-authored, every fact traceable to confirmed prior sessions) specified a native b2: restic scheme for the B2 repos: restic -r b2:xzopia-backup:serverN. Running the off-infra-passphrase decrypt drill for real (from S4) surfaced that this fleet's actual credentials layout — AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, no B2_ACCOUNT_ID/B2_ACCOUNT_KEY — makes the b2: form fail with "Account ID is empty"; the working scheme is the S3-compatible endpoint s3:https://s3.eu-central-003.backblazeb2.com/xzopia-backup/serverN. The draft was corrected before commit (folded into 8fe2d52), so no harm shipped — but a "correct-looking, well-sourced" step was still wrong in practice.
Rule: treat any runbook/procedure step involving live-infrastructure specifics — exact paths, URL schemes, command flags, credential layouts — as unverified until it has actually been executed once, even when every fact it's assembled from traces to a confirmed prior session. Prefer to dry-run the step on the real system before committing the runbook; where that isn't possible, mark the step provisional with a "dry-run before trusting" note rather than presenting it as proven. Assembled-from-true-facts ≠ tested-end-to-end.
⤴ Superseded/generalised by #126 (the Sourced-Facts Rule). This is the special case (runbook steps); #126 is the general law (any asserted fact about a live system).
Date: 10 July 2026 (Simon, after the S1 restic failure + reboot)
Five design failures in a single session, each of which would have been caught by one command or one look and none by a review pass:
| Failure | What would have caught it |
|---|---|
DR runbook b2: restic scheme wrong |
grep RESTIC_REPOSITORY /etc/xzopia-backup/credentials |
| Cowork rejects WSL/UNC paths | trying it once |
| Power Automate can't post as a user to a 1:1 chat | opening the dropdown |
| CIPP New-Invite has no tenant selector | opening the page |
| restic 0.12.1 on S1 (couldn't write the Object-Lock bucket) | restic version on each box |
A design-stage "reliability %" gate was proposed and rejected: there is no honest way to score a design — the 3 Jul Object-Lock cutover would have scored 95%+ (sound reasoning, real evidence, validated on a live box) and still broke S1 four days later. A required number creates pressure to produce it, and a plausible 96% is a false anchor worse than no number. An "AI council" was rejected as the primary mechanism too: multiple LLM reviewers share correlated blind spots — three reviews of that cutover would each have reasoned fluently about restic's checksum behaviour, and none would have asked "what version is installed on S1?".
Rule: No design, plan, or runbook may assert a fact about a live system that has not been read from that system. Every such fact carries its source — the command run, the page observed, the file read, the date. Unverified facts are marked ASSUMPTION and verified before anything is built on them. When correctness depends on a tool's version/path/flag/config, enumerate it on every affected host (see #A/#127). Fluency is not evidence — the source citation is what distinguishes confidently-right from confidently-wrong. Adversarial review is valuable for reasoning and sequencing (blast radius, exceptions, ordering, what-if-wrong) as a separate non-numeric gate — never as a score. Now a standing law in docs/session-protocol.md. Supersedes #125 (a special case of this).
Date: 10 July 2026
The 3–6 Jul B2 Object-Lock cutover was validated on S4 (restic 0.16.4 — writes to the locked bucket fine) and declared complete for the fleet. But the bucket's default retention now requires a PutObject checksum header that restic 0.12.1 on S1 doesn't send — so S1 silently stopped backing up the first night the retention was active, and stayed broken for four days. S2/S4 (0.16.4) and S5 (0.14.0) were unaffected because they were new enough. A "representative box" test hid a version-dependent failure on the one box that was the exception.
Rule: when a change's correctness depends on a per-host tool/version/config, verify it on every affected host, not one. Enumerate the dependency across the fleet (for h in …; do ssh $h 'restic version'; done) before calling the cutover done. This is the concrete, fleet-scoped form of the Sourced-Facts Rule (#126).
Date: 10 July 2026
restic is installed outside any package/config management — /usr/local/bin/restic (hand-placed) shadowing the distro /usr/bin/restic. No policy, playbook, or inventory owns its version, so the fleet drifted to three different versions with no one noticing: S1 0.12.1 (Ubuntu 22.04 apt), S5 0.14.0, S2/S4 0.16.4 (sourced 10 Jul). The 0.12.1 box then failed against a bucket feature the others handled. The same gap applies to any hand-installed binary (restic, mcp-proxy, etc.).
Rule: maintain a third-party binary inventory (name, expected version, install path, which boxes) and review it during the monthly patch window — see docs/runbooks/fleet-patch-reboot-policy.md. A binary that lives in /usr/local/bin and is owned by nobody is a latent fleet-consistency failure waiting for a dependency to change under it.
Date: 10 July 2026
S1's backup failed every night 7–10 Jul. The Mailgun failure alert fired each night (smtpstatus=250 — delivery worked). Yet it went unacted-on for four days until Simon happened to notice on 10 Jul. Delivery is not detection: an alert channel that produces mail no one reads/triages is decorative. The mechanism that eroded this channel is #131 (below) — the backup service throws a spurious failure alert on every reboot, so the channel had been crying wolf, and a real four-day outage blended into the noise.
Rule: an alert only counts if a failure changes someone's behaviour. Two requirements: (1) it must be seen — route actionable alerts to a channel that gets triaged (the severity-tiered routing in docs/design/alert-routing.md), not just an inbox; (2) it must not cry wolf — eliminate false positives (see #131) or the real ones get ignored. A dead-man's-switch (healthchecks.io) that pages on absence is stronger than a mail-on-failure that depends on someone reading it.
Date: 10 July 2026 · ⚠️ ROOT CAUSE SUPERSEDED 20 July 2026 (see below + docs/runbooks/disaster-recovery.md S1 section).
Rebooting S1 for a kernel update left it stuck for ~1h50m — network-reachable (TCP RST) but no services, recoverable only via the Fasthosts console (unreachable over SSH). The durable, still-correct lesson is the operational one: keep console/panel access ready before rebooting any box you can only reach over SSH, because a boot that stalls in firmware is invisible and unfixable from SSH.
❌ Original root-cause theory (10 Jul — now KNOWN WRONG): "a stray Ubuntu ISO in the virtual DVD + BIOS boot order set DVD-first; fix = detach the ISO and set HDD-first." Live console testing on 20 Jul (ticket UK1521645557) disproved this.
✅ CONFIRMED root cause (20 Jul, reproduced live): S1's VM firmware switches type based on whether virtual media is attached — ISO attached → OVMF/UEFI (boots); ISO detached → legacy SeaBIOS, which cannot boot the UEFI-only disk and hangs forever on "Booting from Hard Disk...". So for S1 specifically the ISO must stay PERMANENTLY attached — detaching it is what breaks the boot (the exact opposite of the original "detach the ISO" advice). Confirmed recovery = keep the ISO attached, then at the GRUB menu select "Boot from next volume" (twice — attempt 1 Load Errors, attempt 2 boots the disk). Full mechanism + recovery steps: docs/runbooks/disaster-recovery.md (S1 section). The generic "check boot order / no stray ISO" pre-reboot check still applies to S2/S4/S5 — but NOT to S1, which is the media-attach exception.
Date: 10 July 2026
xzopia-backup.timer is Persistent=true, so on boot systemd fires xzopia-backup.service to catch up — before the data sources are ready. On S1 it ran at 09:31 (up <1 min) and failed instantly: pg_dump: connection to socket …/.s.PGSQL.5432 failed: No such file or directory (Postgres not up yet). On S2 it ran before the Docker containers were healthy. Each time it emits a failure exit + a Mailgun alert — a false alarm on every reboot. Re-running it once dependencies were up succeeded immediately (restic/Object-Lock were never the problem).
Rule: order the backup service After= its data sources (postgresql.service / docker.service) and/or add a readiness wait, or set the timer Persistent=false so it doesn't fire on boot. This is the mechanism behind #129 — an alert channel that cries wolf on every reboot trains everyone to ignore it, so the real four-day S1 outage went unseen. Fix prepared fleet-wide (not yet deployed — pending S1 return); per #127, validate on every box.
Date: 17 July 2026
Adding create_client/create_contact to the ITFlow MCP (S4:3008), sourced against ITFlow's own api/v1/*/create.php + create_output.php on S2. A declined insert — a required field missing, or a duplicate contact email within a client — does not return an HTTP error. create_output.php returns HTTP 200 with {"success":"False","message":"…insert query failed…"} in the body. The MCP's itflowPost() only throws on !res.ok, so a failed create was being returned to the caller as a success. The pre-existing create_ticket/close_ticket tools have the same latent gap.
Rule: for any ITFlow (or similar PHP-API) write tool, inspect the response body's success flag, not just the HTTP status — treat success:"False" as an error and surface the server message. Implemented as an assertInsertOk(r) guard on the new tools; verified live (a deliberate duplicate contact correctly surfaced isError:true). Worth back-fitting the same guard onto create_ticket/close_ticket.
Date: 17 July 2026
Cleaning up a throwaway verification client revealed two ITFlow API facts that matter for the upcoming client-import/reconciliation: (1) there is no clients/delete.php — the only API-supported lifecycle op is clients/archive.php (soft archive, sets client_archived_at); a true hard-delete needs the ITFlow web UI or direct DB access. Contacts do have a contacts/delete.php (hard delete). (2) clients/read.php (→ the list_clients MCP tool) does not filter archived clients — an archived client still appears in list_clients with no obvious archived-vs-active distinction unless you read client_archived_at.
Rule: the client-reconciliation/import logic must not treat list_clients as an active-only view — filter on client_archived_at IS NULL itself, and expect to hard-delete stray records via the UI/DB, not the API. Flag any leftover archived test/dummy clients for manual UI deletion before a bulk import so they don't collide with real records.
companyId, not a nested company:{id} relationDate: 17 July 2026
The twentycrm MCP's create_person linked a person to a company by sending body.company = { id: company_id }. Twenty's REST API rejected every company-linked create with 400 BadRequestException — "Relation company...". The person object's actual foreign key is the scalar companyId (visible in list_people output). Fix: body.companyId = company_id. Bug had been latent because the seed/demo data was inserted directly, and no prior tool call had linked a person to a company. Found during the client-import pilot; would have blocked the whole import.
Rule: for Twenty CRM REST writes, set relations as scalar *Id fields (companyId), not nested {relation:{id}} objects. When adding a write tool, exercise the relation path once — the field name that works for reads (company) is not the one the write API expects.
Date: 17 July 2026
The pilot import fired rapid create_contact POSTs at ITFlow (psa.xzopiasecure.com) and got HTTP 403 after ~3 requests ("the API key may lack scope" — a misleading message; the key was fine). Root cause: S2's Apache has mod_evasive (evasive20) enabled, which blocks a client that exceeds ~2 requests to the same URI per second (DOSPageCount 2 / DOSPageInterval 1), for a DOSBlockingPeriod 600 — a 10-MINUTE block, not ~10s (live-verified /etc/apache2/mods-enabled/evasive.conf, 19 Jul 2026; an earlier draft of this lesson said "~10s" — corrected). DOSWhitelist 127.0.0.1 ::1, so local / docker exec checks are exempt. Not fail2ban (S2's only jail is sshd; the source IP was never banned — reads kept returning 200). Both ITFlow and Twenty CRM sit behind this same Apache, so both write paths are affected.
Rule: any bulk write against an S2-hosted API (ITFlow, Twenty) must throttle to ≤1 request/sec per endpoint (the pilot used a 1.3s inter-write sleep + a 12s back-off-and-retry on a 403) — or temporarily whitelist the source IP via DOSWhitelist for the import window. For the full ~68-org / ~600-contact import, budget for the throttle (≈20-40 min per system) or use the whitelist. Also make importers resume-safe (skip already-present records by email) so a mid-run 403 doesn't force a restart.
Date: 19 July 2026
Report id=139 marked ITFlow's colour branding "done" after a direct curl of /css/itflow_custom.css returned the brand block — which was true, and stayed true. But chat later fetched the same URL and got 901 bytes of stock CSS, and the live nav still showed AdminLTE orange. Root cause: browser heuristic caching. ITFlow's <link href="/css/itflow_custom.css"> has no cache-buster and Apache sent no Cache-Control/Expires; the pre-change file's Last-Modified was weeks old, so browsers cached the old CSS with a long heuristic freshness window and never revalidated. The file + origin were correct the whole time (verified from 3 vantages incl. an external network); the change simply wasn't reaching already-cached browsers. Logo + favicon were fine (freshly set, no prior cache).
Rule: for server-edited CSS/JS/assets with no URL cache-buster, "done" requires verifying propagation, not just origin bytes — fetch with Cache-Control: no-cache AND set a revalidation header. Fix applied: an Apache <Location "/css/itflow_custom.css"> Header set Cache-Control "no-cache, must-revalidate" on the psa vhost (durable, outside ITFlow's git tree), so browsers always revalidate via ETag. Also tell the user to hard-refresh / use incognito once to clear an existing stale cache. (Full analysis: cc-report id=144.)
Date: 19 July 2026
During the WordPress media extraction, AI-flagged images were re-checked for genuine Adobe Stock licensing via exiftool. Two legitimately-licensed images (asset IDs 597622133, 599990817, contributor "RealPeopleStudio") were nearly mis-excluded because an exact string-match on stock.adobe.com failed — the IPTC Credit field (2:110) is limited to 32 bytes, so "RealPeopleStudio - stock.adobe.com" (34 chars) is stored truncated as "RealPeopleStudio - stock.adobe.c". The XMP copy isn't length-limited, but exiftool surfaced the truncated IPTC value. Also: WordPress strips metadata from -scaled derivatives, so a licence check must read the ORIGINAL upload (media_details.original_image), not the display image.
Rule: when matching embedded licence provenance, match a partial marker (stock.adobe, not stock.adobe.com) to survive IPTC's 32-char Credit truncation, and read the original file (not the WP -scaled derivative, which is metadata-stripped). Result held: 12 of 16 AI-flagged images confirmed Adobe Stock licensed and kept; 4 with no licence metadata excluded. (cc-report id=146.)
Date: 19 July 2026
SSH to S3 (79.99.41.247) for the Comet investigation was blocked — but it was never an SSH/network error. Claude Code's permission layer rejected any Bash command containing that IP before it ran, via a deny rule Bash(* 79.99.41.247 *) in .claude/settings.json — a leftover from S3's old decommission-MCP role. Simon spent time checking UFW / fail2ban / hosts.deny / hosts.allow / authorized_keys / sshd_config on S3, all clean, because the box was never contacted. The tell: no "Connection refused / timed out / Permission denied" from SSH at all — the harness denial message names the exact deny rule.
Rule: when SSH/commands to a specific host fail with no server-side SSH error and every server-side control checks clean, look at the client-side permission config (.claude/settings.json permissions.deny) for a stale host/IP pattern before debugging the server. Removed the rule (commit 3df3f1a); SSH then worked first try. Keep host-scoped deny rules in sync with a server's current role. (cc-report id=145.)
resolve_threat silently no-ops unless an analyst verdict is set firstDate: 22 July 2026
A bulk S1-EDR threat clean-up appeared to succeed — resolve_threat returned without error — but the threats stayed unresolved. Root cause: SentinelOne's incident API will not transition a threat to resolved while its analystVerdict is undefined (unset); the resolve call is accepted and silently no-ops rather than erroring, so nothing in the response signals the failure. The threats had never been triaged, so their verdict was still undefined. A second gotcha surfaced in the same batch: resolving in one large sweep hit a ~80-item ceiling — beyond roughly 80 threats the batch quietly stops applying.
Rule: when bulk-resolving S1-EDR threats via the SentinelOne MCP, set_analyst_verdict first (e.g. true_positive / false_positive) and only then resolve_threat — a resolve on an analystVerdict:undefined threat is a silent no-op, not an error. Chunk batches to ≤~80 items and re-read each threat's state after resolving to confirm it actually flipped (don't trust the empty/OK response). Verify, don't assume the call worked. (XS item 5; originally flagged in session-handoff id=15, 8 Jul.)
?ids= collections, others are /{scope}/{id}/sub-resource; verify eachDate: 23 July 2026
Two S1-EDR MCP tools shipped with the wrong endpoint shape, in opposite directions — both 404'd live. get_agent_passphrase assumed a per-agent sub-resource (/agents/{id}/passphrase) when the passphrase is on a filtered collection (/agents/passphrases?ids={id}) — lesson #122. list_policies assumed a collection (/policies) when a policy is a per-scope single object (/accounts|sites|groups/{id}/policy) — XS item 58, this lesson. The two bugs are mirror images: guessing "collection" and guessing "sub-resource" each failed once. The list_policies bug also silently blocked item 57's investigation until the correct endpoint was called directly.
Rule: never infer a SentinelOne v2.1 endpoint's shape from a sibling tool — the API mixes ?ids= filtered collections and /{scope}/{id}/sub-resource reads with no consistent pattern. Check each endpoint against the API reference (or probe it live: a dummy-id 404 vs 400/200 distinguishes "route doesn't exist" from "route exists, bad param" — the technique used to confirm both fixes without exposing data). When adding or reviewing an S1 connector tool, confirm the exact path exists before shipping; a repo fix isn't live until deployed to S4 (backup-first) and re-verified end-to-end through the connector, not just by curl. (XS items 8/58; both deployed + verified 23 Jul.)