The health check returns 200 but the service is down

A health check that returns 200 without touching its dependencies proves exactly one thing: the HTTP process is up. The database can be stopped, the connection pool exhausted, the service is down for every real request, and the dashboard stays green. An honest check runs a real query against each dependency and keeps the latency of every probe, because latency is what drifts first when a system saturates.

What does a 200 actually prove?

That a process is listening on the port, read your request, and wrote three bytes back. Nothing else. To show it, we reproduced the classic failure with a forty-line Node application: a /healthz route that answers ok without asking anyone anything, and an /orders route that counts the rows of a PostgreSQL table through a pool of ten connections.

$ curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" http://127.0.0.1:8200/healthz
200 in 0.002964s
$ curl -s http://127.0.0.1:8200/orders
{"orders":100000}

Stop the database, then ask the exact same two questions again:

$ curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" http://127.0.0.1:8200/healthz
200 in 0.000593s
$ curl -s -w "-> %{http_code} in %{time_total}s\n" http://127.0.0.1:8200/orders
error: connect ECONNREFUSED 127.0.0.1:5432
-> 500 in 0.001239s

The dashboard polling /healthz shows the service green while every client gets an error. Both readings are accurate; they measure different things.

What does a deep check change?

Everything, and it fits in eight lines: the same question, asked of the dependency instead of the process.

+if (req.url === '/health/deep') {
+  try {
+    await pool.query('select 1')
+    res.writeHead(200); res.end('ok\n')
+  } catch (e) {
+    res.writeHead(503); res.end(`degraded: ${e.message}\n`)
+  }
+  return
+}

With the database running, then stopped:

$ curl -s -w "-> %{http_code} in %{time_total}s\n" http://127.0.0.1:8200/health/deep
ok
-> 200 in 0.003124s
$ curl -s -w "-> %{http_code} in %{time_total}s\n" http://127.0.0.1:8200/health/deep
degraded: connect ECONNREFUSED 127.0.0.1:5432
-> 503 in 0.000930s

Each level of checking proves a little more, and none of them proves everything:

The last row of that table is the rest of this article: even a deep check, read as a boolean, arrives late.

Why does latency move before the status code?

Because between "fine" and "error", most systems pass through "waiting". A connection pool, a request queue, a saturated disk: demand exceeds capacity, waits stretch, and the status code only flips once an explicit limit is reached, almost always a timeout.

We measured it on the same application. A stream of heavy requests saturates the service little by little: each one holds a pool connection for one second, twelve of them arrive per second against a pool of ten, and a probe hits /orders every two seconds.

Thirty seconds of drift and eight green probes before the first error code.

The probe climbs from 10 ms to 4.6 s in regular steps, returning 200 from the first reading to the eighth. The first 500 only lands in the thirty-second second, when waiting for a connection exceeds the pool's limit, five seconds here, and its body no longer looks like a stopped database. Caught live, during the saturation: the shallow check sees nothing, the real route hangs for five seconds, and the request over the limit gets the error.

$ curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" http://127.0.0.1:8200/healthz
200 in 0.001059s
$ curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" --max-time 30 http://127.0.0.1:8200/orders
200 in 4.920710s
$ curl -s -w "-> %{http_code} in %{time_total}s\n" --max-time 30 http://127.0.0.1:8200/orders
error: timeout exceeded when trying to connect
-> 500 in 5.001503s

The opposite is also true, and it needs saying: sudden death gives no warning. When we stopped the database outright, the refusal arrived in three milliseconds with no drift before it at all. Latency announces the failures that build up, not the ones that drop.

How many probes does it take to see the drift?

Enough for the drift to fit inside them, and our incident gives the scale. Thirty seconds of warning with one probe every thirty seconds is a single reading: one abnormal point, indistinguishable from a network hiccup. With a probe every two seconds it is eight points in a row, and nobody mistakes eight rising steps for chance.

So you need two things the status code does not give you: the latency of every probe, kept, and a window long enough to read a trend in it. A slowness threshold then turns the curve into a signal: at 800 ms, our incident declares itself on the third probe, twenty-seven seconds before the first 500. Zeal in the other direction has its own trap: declaring an outage on a single failed probe pages someone for every hiccup, and two consecutive failures make a much better rule.

The Health section of Kestro applies exactly these settings: one measurement per service every thirty seconds by default, latency kept probe by probe with a "slow" state past 800 ms, an outage declared on the second consecutive failure only, and an expected word in the body to catch error pages served as 200s. A curl loop in a terminal does the same job if you would rather not add a tool. And the lying green light is not specific to HTTP: an SSH tunnel survives a Mac's sleep with a live process and a dead connection, which is the same lesson.

What remains

These outputs come from a Linux x86_64 machine on 24 August 2026: Node v22.21.1, PostgreSQL 16.15 in Docker (image postgres:16), pg driver 8.23.0, curl 8.14.1. The curve's numbers depend on choices we made: the heavy requests last one second because we decided so, and the first 500 falls at five seconds because that is the pool's configured connection timeout. The shape of the curve travels; the thirty seconds do not.

The warning assumes a failure that builds. A database killed outright, an unplugged cable, a process taken by the out-of-memory killer leave no drift to read, and our own first session shows it.

Finally, a deep check has a cost we did not measure here: it runs a real query on every probe, against the very dependency you are trying to spare, and ours shares the application's pool, so it queues with the application. At what interval does that cost become a problem? We do not know yet, and it is the question to ask before multiplying probes.