Monitoring, Observability and Alerting

The Risk

Your application logs only show requests that actually reached your app. They are blind to everything else: what the CDN blocked at the edge, a request that timed out before it ever arrived, a disk quietly filling up, or a TLS certificate about to expire. It is like watching only the front door while ignoring the windows, the smoke alarm, and the fuel gauge. Real example: an edge timeout never reaches your server, so your app log shows nothing at all while real users are getting errors.

The Solution

Watch each layer with its own lens and accept that no single log has the whole picture. The layers that matter for most setups: the CDN edge stream (every request the world sends you, including the ones that got blocked), per-API request logs, host and infrastructure health (CPU, memory, and especially disk), TLS certificate expiry, and read-path liveness of each public API. Each one catches a failure the others cannot see.

The Fix

# The layers to watch, and the failure each one uniquely catches:
#
#   Edge / CDN stream    -> blocked probes, scanners, timeouts that never reach origin
#   API request logs     -> slow endpoints, error spikes, unknown paths being probed
#   Host / infra         -> disk filling, CPU/RAM exhaustion (NO http signal at all)
#   TLS cert expiry      -> a cert about to lapse (silent until the site goes down)
#   Read-path liveness   -> the actual feature is down even though the app is "up"
#
# One lens per layer; wire them all into one place (see the dashboard item).

The Risk

If each signal lives in a different tool - the hosting console for CPU, the CDN dashboard for traffic, a database console for queries, SSH for disk - you will check none of them regularly and miss the early warning. Scattered monitoring is monitoring you do not actually do.

The Solution

Funnel every signal into ONE dashboard you can scan in seconds. Put a status-at-a-glance board on top (a simple green / amber / red tile per system: APIs, databases, frontends, infra, security) so the question "is anything on fire right now" is answered instantly, with drill-down into each area behind it. The value is having a single place to look, not the specific layout.

The Fix

# Top level: one tile per system, colored by its worst current state.
#
#   [ APIs  OK ]  [ DBs  OK ]  [ Frontends  OK ]  [ Infra  WARN ]  [ Security  OK ]
#
# - Green = healthy, Amber = warning, Red = needs attention now.
# - Each tile links to the detailed view for that area (the drill-down).
# - Refresh on an interval; show "updated Ns ago" so stale data is obvious.
# Reactive by design: this is what you CHECK. It does NOT replace alerting
# (the thing that reaches you when you are not looking - see the alerting item).

The Risk

A shallow /health endpoint often returns OK even when the actual feature is broken. A health route that just says "the app process is running" will happily report healthy while the database read path behind it is dead. It is a shop with the OPEN sign lit but the till broken: looks fine from outside, sells nothing.

The Solution

Probe the ACTUAL endpoint the way a user would, and check for a known marker in the response, not just a 200. If your API returns data, the probe should ask for data and confirm the expected content came back. That is the difference between "the server answered" and "the feature works".

The Fix

# Bad: pings a route that is always up
#   curl -sf https://api.example.com/health        # 200 even if the DB is down
#
# Good: exercises the real path and checks a marker in the body
#   curl -sf https://api.example.com/query -d 'SELECT 1' | grep -q '"result"'
#   # non-zero exit -> alert. The probe fails when the FEATURE fails, not just
#   # when the process dies.
#
# Run it on a schedule from OUTSIDE the box (a cron, an uptime service) so you
# also catch network / edge failures, not only app crashes.

The Risk

Monitoring only helps if you are looking at it, and at 3am you are not. Without alerting you find out about an outage from an annoyed user hours later, or from your own dashboard the next morning. A dashboard is passive; you need something that actively reaches you.

The Solution

Treat alerting as a SEPARATE system from monitoring - its whole job is to reach you when you are NOT watching. Route alerts to a push channel and / or email, tier them by severity (a warning that escalates to critical only if it persists, so you are not paged for a one-second blip), and de-duplicate so one incident is one alert, not fifty. Keep it practical: TIGZIG uses Pushover for phone push notifications (a simple, low-cost service with a small one-time fee per platform) and Brevo for transactional email. Many alternatives work just as well - email via Amazon SES / Postmark / Resend, push via ntfy or Pushover, chat via a Slack or Discord webhook, on-call paging via PagerDuty or Opsgenie - choose by your scale and budget; the pattern is identical. A safe, concrete example: a public API that stops responding for a sustained short window raises an alert to your phone.

The Fix

# Alerting is a thin layer your monitors call when a check crosses a line.
# Illustrative helper (channel-agnostic - swap in whatever you use):
#
#   def alert(title, message, severity):        # severity: 'warning' | 'critical'
#       key = dedup_key(title)                   # e.g. hash of service + check
#       if already_active(key):                  # one incident = one alert
#           return
#       mark_active(key)
#       push_notify(title, message, priority=severity)   # e.g. Pushover
#       if severity == 'critical':
#           send_email(title, message)                   # e.g. Brevo / SES
#
#   # on recovery: clear the dedup key and send a "resolved" ping.
#
# Warning first, escalate to critical only if it persists, so a transient
# blip does not wake you.

The Risk

Users and integrators want to know if you are up, so a public status page is genuinely useful. But wiring that page directly to your production database to read live health puts your database one query away from the whole internet. The convenience becomes an attack surface.

The Solution

Use a one-way bridge so the public page can never touch your data. A private job writes a small summary file (per-service up or down, plus rolling uptime) to object storage; the public page reads ONLY that file. The browser never reaches your database or your logging service. It is a notice board out front that a staff member updates, versus handing visitors the keys to the server room.

The Fix

# One-way flow - the browser can only ever see the summary file:
#
#   [private cron] --writes--> status.json (object storage) <--reads-- [public page]
#         (inside)                                                    (no DB access,
#      has DB access                                                  no logger access)
#
# - The cron aggregates health and overwrites a tiny JSON on a schedule.
# - The public page (or an edge worker) renders that JSON server-side.
# - There is NO code path from the public page back to the database.

The Risk

Automation only catches what you told it to check, so a brand-new failure mode or a novel probe slips straight through. And a dashboard that nobody actually looks at is not monitoring - it is decoration.

The Solution

Run BOTH tiers. First, a routine automated sweep on a schedule that confirms your health and posture checks are green. Second, a deliberate human review that looks for what the automation missed. Treat observation as a control in its own right, not an afterthought. A concrete example: a short daily pass that eyeballs the traffic no rule flagged, specifically to spot the slow, quiet, low-volume probe that a threshold would never trip.

The Fix

# Tier 1 - automated (scheduled): pass/fail on the things you already know to check
#   * are all monitors reporting? any red tiles? any check stale / not run?
#   * any new blocked-source spike, error spike, or cert nearing expiry?
#
# Tier 2 - human (daily, a few minutes): look for the KNOWN-UNKNOWNS
#   * scan the traffic that matched NO rule - what does not fit?
#   * anything suspicious that is not yet a rule? note it, decide if it becomes one.
#
# The automated tier keeps the floor; the human tier finds tomorrow's rule.

The Risk

A new blocking rule that is tuned wrong will block real users: a rate limit set too tight, a scanner filter that also matches a legitimate crawler, a velocity rule that trips on a busy real customer. Shipping it straight to "enforce" risks a self-inflicted outage that looks exactly like an attack.

The Solution

Run every new blocking, velocity, or rate rule in observe-only mode first. Log what it WOULD have blocked, review that list with your own eyes, confirm no legitimate traffic is caught, and only THEN flip it to actually enforce. It is testing the smoke alarm before you wire it to the sprinklers.

The Fix

# Ship the rule in two stages:
#
#   Stage 1 (observe): if rule_matches(request): log_candidate(request)  # do NOT block
#   ...review the candidate list. Any real users / good crawlers in there? Re-tune.
#   Stage 2 (enforce): if rule_matches(request): block(request)          # now act
#
# Keep the observe path available even after enforcing, so you can drop back
# to observe instantly if something legitimate starts getting caught.