Incident Response

The Risk

The instinct during an attack is to restart the container, redeploy, or block at the firewall. Each of those destroys something you need. A restart or redeploy throws away the container logs. A firewall block means the next request never arrives, so you record nothing about what they try next. And a 403 right after a payload tells the attacker two things: that it worked, and that you are watching. Blindly preserving logs while data is being taken is also wrong. The real question is where in the chain you can stop the harm while still writing everything down.

The Solution

Build a stop lever that sits AFTER logging and classification but BEFORE execution. The simplest form is a flag file on a mounted path that the request handler checks late: if the flag is up, the request is still received, still logged, still classified, and then returns a plain 503 instead of running. No restart, no deploy, no lost evidence, and the attacker sees an outage rather than a verdict. Set the flag through a small script that records who set it and why, because the reason is the first thing the next person needs. Then assess with no time pressure, because nothing is executing. A block at the edge is the fallback for a platform-wide problem, not the first move, because it records nothing of yours. Capture logs before anything that recreates the container. Stop the container almost never.

The Fix

# The order, cheapest-and-still-recording first:
1. FLAG     raise the lockdown flag via the script (records who / why / when).
            Requests keep landing in your logs; they just do not execute. Returns 503.
2. ASSESS   read what arrived while flagged. No clock is running - nothing executes.
3. EDGE     block at the CDN ONLY if the whole platform is at risk (it records nothing of yours).
4. CAPTURE  save container logs + request logs + CDN analytics BEFORE any restart or redeploy.
5. FIX      patch, deploy, lift the flag, watch the first hour closely.
6. STOP     the container - almost never; it is the step that costs the evidence.

# In the request path (any framework):
#   log(request); band = classify(request)
#   if flag_is_up(): return Response(503)     # after the log, before the execute
#   execute(request)

The Risk

A secret committed to git - even briefly, even in a private repo - should be considered compromised. Bots that scrape GitHub for secrets operate within minutes. Simply deleting the file doesn't help because the secret remains in git history forever unless explicitly scrubbed.

The Solution

The moment you discover a leaked credential, generate a new one immediately - don't wait to investigate first. Update the new credential in all your environments (Vercel, Coolify, local), then revoke the old one from the provider's dashboard. Check the provider's access logs to see if anyone used the leaked credential while it was exposed. If the repo might go public, scrub the secret from git history too.

The Fix

1. Rotate the credential IMMEDIATELY (generate new)
2. Update env vars in all environments
   (Vercel, Coolify, local)
3. Revoke the old credential from provider dashboard
4. Check git history for other exposures:
   git log --all --oneline -- "**/.*env*"
5. Audit provider access logs for unauthorized usage
   during the exposure window

# Scrub from git history if repo may go public:
# Use BFG Repo-Cleaner or git filter-repo

The Risk

Most teams only rotate credentials after a breach. But secrets accumulate risk over time - they get shared in Slack, copied to local machines, cached in CI pipelines, and stored in browser password managers. The longer a secret lives, the more places it exists and the more likely it has leaked somewhere you don't know about.

The Solution

Have a documented plan for how to rotate each credential your app depends on: database passwords, API keys, OAuth client secrets, signing keys. Know the steps BEFORE you need them - during a breach is the wrong time to figure out the process. Ideally, rotate proactively on a schedule (quarterly for high-value secrets). At minimum, know which environment variables need updating when a key changes, and test that your app handles the rotation without downtime.

The Fix

# Secret rotation checklist per credential:
# 1. Where is it stored? (Vercel, Coolify, .env, CI)
# 2. What breaks if I change it? (which services)
# 3. Can I rotate without downtime?
#    (some services support two active keys)
# 4. How do I generate a new one?
#    (provider dashboard, CLI, API)
# 5. What environments need updating?
#    (prod, staging, local, CI)

# Practical steps:
# - Document rotation procedure for each key
# - Keep a list: which secrets exist, where, last rotated
# - After any team member leaves: rotate shared secrets
# - After any security incident: rotate everything

The Risk

When an incident happens, your first instinct is to SSH into the affected container and grep logs. With 30+ backends, this takes hours - and by the time you find the relevant entries, the attacker may have moved to another service. Container logs are also lost on redeploy. If you don't have centralized logging set up BEFORE the incident, you're doing forensics blind.

The Solution

Your centralized API logging dashboard (item 2.14) is the first place to go during any incident. Filter by the suspicious IP across ALL backends to see the full attack timeline - what they probed first, which endpoints they hit, what payloads they sent. The 30-day window of request bodies and client IPs gives you the raw evidence. Cross-reference with Cloudflare analytics (which shows blocked requests the backend never saw) and pg_stat_activity (for active database queries). This is why centralized logging must be set up before you need it - during an incident is too late.

The Fix

# Incident investigation sequence:

# 1. Centralized API dashboard - filter by suspicious IP
#    See: all endpoints hit, payloads sent, status codes, timing
#    Cross-backend: did this IP probe other services?

# 2. Cloudflare analytics - what was BLOCKED
#    Dashboard > Analytics > filter by IP
#    Shows requests that never reached your backend
#    (WAF blocks, rate limit blocks, challenge failures)

# 3. Database activity - active/recent queries
#    SELECT * FROM pg_stat_activity WHERE state = 'active';
#    Check for: long-running queries, unusual query patterns

# 4. Container logs - only if centralized logs are incomplete
#    docker logs <container> --since 2h | grep <ip>
#    WARNING: lost on redeploy, this is your last resort

# 5. Timeline reconstruction
#    Combine all sources into a single timeline:
#    IP → first probe → escalation → data access → blocking