SPRINT: EPSILON — Forensic Analysis: Breach Detection & Backdoor Sweep

Issued by: Omega v2.7 | Date: April 13, 2026 Priority: 🔴 EMERGENCY SECURITY Builder: Epsilon Time: 2-3 hrs per server (ghost-01, cx23, cx23-b) Context: Active brute force attack discovered on ghost-01 (45.148.10.147). cx23-b found with UFW inactive, PasswordAuth=yes, RootLogin=yes, no fail2ban — completely exposed. Must determine if any server was compromised.

---

CRITICAL CONTEXT

SSH password auth was ON with root login allowed on at least ghost-01 and cx23-b. Brute force bots hammer every Hetzner IP continuously. If any bot succeeded, they could have: - Installed crypto miners (most common) - Planted SSH backdoors (authorized_keys, SUID binaries) - Installed rootkits (kernel modules, LD_PRELOAD hooks) - Added cron jobs for persistence - Modified system binaries (replaced ssh, sshd, ls, ps, netstat with trojanized versions) - Exfiltrated credentials (.env files, API keys, database passwords) - Planted web shells in nginx-accessible directories - Added new user accounts - Installed reverse shells that phone home on a schedule

Assume compromise until proven otherwise. Check everything.

---

PHASE 1: IMMEDIATE TRIAGE (run first on each server)

1.1 Check for unauthorized users

# List all users with login shells
grep -E '/bin/(bash|sh|zsh|fish)' /etc/passwd
# Expected: root and maybe 1-2 service accounts. Any unknown user = COMPROMISED

# Check for users with UID 0 (root-equivalent)
awk -F: '$3 == 0 {print $1}' /etc/passwd
# Expected: only "root". Anything else = COMPROMISED

# Check for recently added users
ls -lt /home/
# Any unexpected home directories = COMPROMISED

# Check /etc/passwd and /etc/shadow modification times
stat /etc/passwd /etc/shadow
# If modified recently (after server provisioning) = SUSPICIOUS

1.2 Check SSH authorized_keys for backdoors

# Check root's authorized_keys
cat /root/.ssh/authorized_keys
# Should ONLY contain J's public key(s). Any unknown key = COMPROMISED
# Count the keys:
wc -l /root/.ssh/authorized_keys
# If more keys than expected = COMPROMISED

# Check ALL users' authorized_keys
find / -name "authorized_keys" -type f 2>/dev/null
# List content of each:
find / -name "authorized_keys" -type f -exec echo "=== {} ===" \; -exec cat {} \; 2>/dev/null

# Check for authorized_keys in unusual locations
find / -name "authorized_keys2" -type f 2>/dev/null
# authorized_keys2 is sometimes used as a stealth backdoor

1.3 Check for active connections RIGHT NOW

# Who is logged in?
w
who
last -20

# Active network connections — look for unexpected outbound
ss -tnp | grep ESTAB
# Expected: SSH from J's IP, maybe connections to Hetzner mirrors
# SUSPICIOUS: connections to unknown IPs, especially on high ports or to known C2 ranges

# DNS lookups for any suspicious IPs
for ip in $(ss -tnp | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort -u); do
  echo "$ip → $(host $ip 2>/dev/null | head -1)"
done

1.4 Check for running crypto miners (most common post-breach)

# High CPU processes
top -bn1 | head -20
# Any process using >50% CPU that isn't Ollama/nginx/python = SUSPICIOUS

# Known miner process names
ps aux | grep -iE "xmrig|minergate|stratum|cryptonight|monero|kdevtmpfs|kworkerds|dbused|ksoftirqd.*miner" | grep -v grep
# Any matches = COMPROMISED

# Check for processes hiding their name
ls -la /proc/*/exe 2>/dev/null | grep -i deleted
# "(deleted)" entries = process running from a binary that's been removed (hiding tracks)

# Check /tmp and /var/tmp for suspicious binaries
ls -la /tmp/ /var/tmp/ /dev/shm/
find /tmp /var/tmp /dev/shm -type f -executable 2>/dev/null
# Executable files in temp directories = SUSPICIOUS

---

PHASE 2: PERSISTENCE MECHANISMS

2.1 Cron jobs (most common persistence)

# Root crontab
crontab -l 2>/dev/null
# Expected: only our jobs (checkup batch, boxing scraper, cache cleanup)
# Any curl/wget to unknown URLs, base64 decoded commands, or unfamiliar scripts = COMPROMISED

# System crontabs
cat /etc/crontab
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.weekly/ /etc/cron.monthly/
cat /etc/cron.d/* 2>/dev/null

# ALL users' crontabs
for user in $(cut -f1 -d: /etc/passwd); do
  crons=$(crontab -u $user -l 2>/dev/null)
  if [ -n "$crons" ]; then
    echo "=== $user ==="
    echo "$crons"
  fi
done

# Check for cron jobs hidden via @reboot (runs on every boot)
grep -r "@reboot" /etc/cron* /var/spool/cron/ 2>/dev/null

2.2 Systemd persistence

# List ALL enabled services — look for anything unexpected
systemctl list-unit-files --state=enabled | grep -v "^$"
# Expected: ssh, nginx, ollama, ghost-router, sports-cache, checkup-relay, coturn, 
#           fail2ban, unattended-upgrades, and standard Ubuntu services
# SUSPICIOUS: anything with random/obfuscated names

# Recently created service files
find /etc/systemd/system/ /lib/systemd/system/ -name "*.service" -newer /etc/hostname -type f 2>/dev/null
# Shows services created after server provisioning

# Check for user-level systemd services (common hiding spot)
find /root/.config/systemd/ /home/*/.config/systemd/ -name "*.service" 2>/dev/null

# Check for timers (systemd equivalent of cron)
systemctl list-timers --all

2.3 SUID/SGID binaries (privilege escalation backdoors)

# Find ALL SUID binaries
find / -perm -4000 -type f 2>/dev/null | sort > /tmp/suid_current.txt
cat /tmp/suid_current.txt
# Compare against known good list:
# /usr/bin/sudo, /usr/bin/passwd, /usr/bin/chsh, /usr/bin/chfn, /usr/bin/newgrp,
# /usr/bin/gpasswd, /usr/bin/su, /usr/lib/dbus-1.0/dbus-daemon-launch-helper,
# /usr/lib/openssh/ssh-keysign, /usr/bin/mount, /usr/bin/umount, /usr/bin/fusermount3
# ANYTHING ELSE (especially in /tmp, /var, /opt, /dev) = COMPROMISED

# Check for SUID shells specifically (instant root)
find / -perm -4000 -name "*sh" -o -perm -4000 -name "*bash" 2>/dev/null | grep -v "/usr/bin/"
# Any results = CRITICAL COMPROMISE

# Find SGID binaries
find / -perm -2000 -type f 2>/dev/null | sort

2.4 Modified system binaries

# Check if critical binaries have been replaced with trojans
# Compare against package manager's known checksums
dpkg --verify 2>/dev/null | grep -E "^..5" | head -30
# "..5" in column 3 means MD5 checksum changed — binary was modified
# Expected: some config files change legitimately
# SUSPICIOUS: /usr/bin/ssh, /usr/sbin/sshd, /bin/ls, /bin/ps, /bin/netstat, /usr/bin/top, /usr/bin/find

# Check specific critical binaries
for bin in /usr/bin/ssh /usr/sbin/sshd /bin/ls /bin/ps /usr/bin/top /usr/bin/find /usr/bin/curl /usr/bin/wget; do
  if [ -f "$bin" ]; then
    echo "=== $bin ==="
    stat "$bin" | grep -E "Modify|Change"
    file "$bin"
    md5sum "$bin"
  fi
done

# Check if LD_PRELOAD is being used to intercept system calls
echo $LD_PRELOAD
cat /etc/ld.so.preload 2>/dev/null
# Should be empty. Any content = ROOTKIT

2.5 Kernel module rootkits

# List loaded kernel modules
lsmod | sort
# Compare against expected Ubuntu modules
# SUSPICIOUS: modules with random names, single-letter names, or names mimicking legitimate modules

# Check for hidden kernel modules (some rootkits hide themselves from lsmod)
ls /sys/module/ | sort > /tmp/sysmodules.txt
lsmod | awk '{print $1}' | sort > /tmp/lsmod.txt
diff /tmp/sysmodules.txt /tmp/lsmod.txt
# Significant differences = module hiding = ROOTKIT

# Check kernel log for module loading
dmesg | grep -iE "module|insmod|modprobe" | tail -20
journalctl -k | grep -iE "module" | tail -20

---

PHASE 3: CREDENTIAL EXPOSURE

3.1 Check if our secrets were accessed

# Check access times on sensitive files
stat /opt/ghost/airport_pipeline/.env
stat /root/.secrets/cf_api_token 2>/dev/null
stat /etc/systemd/system/checkup-relay.service
stat /etc/systemd/system/sports-cache.service
# If access times are suspicious (accessed when no one was working) = POSSIBLE EXFILTRATION

# Check bash history for suspicious commands
cat /root/.bash_history | grep -iE "curl.*pastebin\|wget.*raw\|base64\|python.*http\|nc\s\|ncat\|/dev/tcp\|scp.*@"
# These patterns indicate data exfiltration or reverse shell activity

# Check if .env files were read recently
find / -name ".env" -type f -exec stat {} \; 2>/dev/null | grep -B5 "Access:"

3.2 Inventory ALL secrets on the server

# Find all .env files
find / -name ".env" -type f 2>/dev/null

# Find files containing API keys or tokens
grep -rl "sk-ant\|sk_live\|SUPABASE\|ANTHROPIC\|TOMTOM\|api_key\|api_token\|secret" /opt/ /etc/ /root/ 2>/dev/null | grep -v ".pyc\|node_modules\|__pycache__"

# If ANY secret was potentially exposed, it needs rotation:
# - TomTom keys (19 accounts)
# - Supabase keys (anon + service_role)
# - Anthropic API key (if set)
# - Brave Search key
# - TURN server shared secret
# - Ghost app token
# - KB Worker secret

---

PHASE 4: NETWORK FORENSICS

4.1 Check for reverse shells or C2 beacons

# Look for processes with network connections to unusual destinations
ss -tnp | grep ESTAB | grep -v ":22\|:443\|:80\|:3478\|:11434"
# Any established connections on unexpected ports = SUSPICIOUS

# Check for listening backdoors
ss -tlnp | grep -v "127.0.0.1\|::1"
# Anything listening on 0.0.0.0 that isn't in our expected list = BACKDOOR

# Check for processes using raw sockets (rootkit communication)
ss -0np 2>/dev/null

4.2 Check DNS for data exfiltration

# Some malware exfiltrates data via DNS queries
# Check for unusual DNS traffic patterns
journalctl -u systemd-resolved --since "7 days ago" 2>/dev/null | grep -iE "NXDOMAIN|SERVFAIL" | wc -l
# Very high count = possible DNS tunneling

4.3 Check iptables for injected rules

# Check for iptables rules that might redirect traffic or hide connections
iptables -L -n -v --line-numbers
iptables -t nat -L -n -v --line-numbers
# Look for rules we didn't create — especially REDIRECT, DNAT, or MASQUERADE rules
# that could be forwarding traffic through the server

---

PHASE 5: FILE SYSTEM ANALYSIS

5.1 Recently modified files (last 7 days)

# Files modified in the last 7 days outside of expected directories
find / -mtime -7 -type f \
  -not -path "/proc/*" \
  -not -path "/sys/*" \
  -not -path "/run/*" \
  -not -path "/var/log/*" \
  -not -path "/var/cache/*" \
  -not -path "/tmp/*" \
  -not -path "/opt/ghost/*" \
  -not -path "/var/lib/*" \
  2>/dev/null | sort
# Review for unexpected modifications to system directories

5.2 Hidden files and directories

# Find hidden files in system directories (common hiding spots)
find /usr /var /opt /etc -name ".*" -type f 2>/dev/null | grep -v ".gitignore\|.env\|.bashrc\|.profile\|.bash_history"
# Hidden files in /usr/bin, /usr/sbin, /var/www = SUSPICIOUS

# Find hidden directories
find / -name ".*" -type d -not -path "/proc/*" -not -path "/sys/*" -not -path "/run/*" 2>/dev/null | grep -v "/\.\|\.cache\|\.config\|\.local\|\.npm\|\.ssh\|\.ollama"

5.3 World-writable files in system paths

find /usr /etc /opt -type f -perm -o+w 2>/dev/null
# World-writable files in system paths = backdoor risk

5.4 Check for web shells

# Common web shell patterns in web-accessible directories
find /var/www /opt/ghost /usr/share/nginx -type f -name "*.php" -o -name "*.asp" -o -name "*.jsp" 2>/dev/null
# We don't run PHP/ASP/JSP — any such file = WEB SHELL

grep -rl "eval\|base64_decode\|exec\|system\|passthru\|shell_exec" /var/www/ /opt/ghost/*.py 2>/dev/null
# Check results carefully — our Python code uses some of these legitimately
# But unexpected files or unexpected occurrences = SUSPICIOUS

---

PHASE 6: AUTH LOG ANALYSIS

# Failed login attempts (last 7 days)
journalctl -u ssh --since "7 days ago" | grep "Failed password" | wc -l
# High count is expected (bots), but look at the breakdown:

# Which usernames were tried?
journalctl -u ssh --since "7 days ago" | grep "Failed password" | awk '{print $(NF-5)}' | sort | uniq -c | sort -rn | head -20
# Expected: root, admin, ubuntu, test, etc. (standard bot wordlists)
# ALARMING: if they tried our actual username or specific service names

# SUCCESSFUL logins (this is what matters)
journalctl -u ssh --since "30 days ago" | grep "Accepted" | head -50
# Every entry here should be from J's IP address
# ANY login from an unknown IP = CONFIRMED BREACH
# Check IPs:
journalctl -u ssh --since "30 days ago" | grep "Accepted" | awk '{print $NF}' | sort -u

# Check for login at unusual times (while J was sleeping)
journalctl -u ssh --since "30 days ago" | grep "Accepted"
# Review timestamps — logins at 3-6 AM in J's timezone (Guadalajara, UTC-6) are suspicious

---

PHASE 7: cx23-b EMERGENCY (highest risk)

cx23-b had UFW inactive + PasswordAuth yes + RootLogin yes + no fail2ban. This is the most likely compromised server.

Run ALL phases above on cx23-b FIRST, before applying any fixes.

If compromise is confirmed: 1. Do NOT just patch it — the attacker may have persistence you haven't found 2. Document everything — screenshot/log all findings 3. Snapshot the disk (Hetzner rescue mode) for forensic preservation 4. Nuke and rebuild — fresh Ubuntu, fresh config, restore only known-good data from backups 5. Rotate ALL secrets that were on that server (TomTom keys, SSH keys, any .env content)

If no compromise found: 1. Apply all security fixes (UFW, SSH hardening, fail2ban) 2. Rotate secrets anyway (they were exposed to password brute force) 3. Monitor closely for 48 hours

---

COMPLETION REPORT TEMPLATE

## Epsilon Forensic Analysis — Completion Report

### Per-Server Summary

#### ghost-01
- Unauthorized users found: [YES/NO — list if yes]
- Unknown SSH keys in authorized_keys: [YES/NO — list if yes]
- Unexpected SUID binaries: [YES/NO — list if yes]
- Suspicious cron jobs: [YES/NO — list if yes]
- Suspicious systemd services: [YES/NO — list if yes]
- Modified system binaries: [YES/NO — list if yes]
- LD_PRELOAD/rootkit indicators: [YES/NO]
- Crypto miner processes: [YES/NO]
- Suspicious outbound connections: [YES/NO — list destinations]
- Successful logins from unknown IPs: [YES/NO — list IPs and times]
- Secrets potentially exposed: [list]
- VERDICT: [CLEAN / SUSPICIOUS / COMPROMISED]

#### cx23
- [Same checklist]
- VERDICT: [CLEAN / SUSPICIOUS / COMPROMISED / UNREACHABLE]

#### cx23-b
- [Same checklist]
- VERDICT: [CLEAN / SUSPICIOUS / COMPROMISED]

### Secrets That Need Rotation
[List every secret that was on a potentially compromised server]

### Fixes Applied
[List every security fix applied during or after the audit]

### Recommendations
[Post-audit security improvements]

### Monitoring Plan
[What to watch for over the next 48 hours]

---

WHAT NOT TO DO

• Do NOT apply fixes on cx23-b before completing the forensic analysis — you'll destroy evidence
• Do NOT assume "UFW blocked it" means safe — UFW can be disabled by an attacker with root
• Do NOT skip the kernel module check — sophisticated attackers hide at the kernel level
• Do NOT ignore successful SSH logins from unknown IPs — that's confirmed breach
• Do NOT just rotate secrets and move on if compromise is found — nuke and rebuild
• Do NOT rush — a thorough 2-hour forensic sweep now prevents months of hidden compromise later