Node.js Event Loop Blocking: Fixing an O(n²) Prefix-Hash Hot Path

A production request can look like an infrastructure problem while the real failure is one synchronous loop. This anonymized Mythos V3 engineering note explains a case of Node.js event loop blocking caused by repeatedly hashing every prefix of a long conversation.

The request body was 5.59 MB and contained 801 history entries that needed a derived lookup key. One application stage occupied the event loop for 28,059 ms. Database and network timings were not the bottleneck; CPU time inside the process was.

Node.js event loop blocking: the symptom

The visible symptom was simple: an otherwise healthy process stopped making progress on one large request. Small requests were fine, resource graphs did not point to a remote dependency, and increasing timeouts would only hide the stall.

We added stage-level timing around request parsing and transformation, plus event-loop delay, process CPU, resident memory, and declared content length. The telemetry intentionally excluded request bodies, credentials, customer identifiers, and generated content. That was enough to isolate one transformation stage without collecting sensitive payloads.

This is the same diagnostic discipline discussed in our regex backtracking CPU-stall post: measure boundaries first, then inspect the narrowest synchronous section.

The accidental O(n²) algorithm

For every target message at index i, the old path created a conversation key from messages.slice(0, i). Creating that key serialized the full prefix and hashed it. Repeating this for hundreds of targets meant the beginning of the conversation was serialized again and again.

For n messages, the total work looked like:

1 + 2 + 3 + ... + n = n(n + 1) / 2

That is quadratic growth. It stays invisible in normal chats and becomes dramatic only when both the history and the number of lookup points are large.

One pass with incremental SHA-256

The replacement computes a stable serialization for each message once, updates a single incremental SHA-256 state as it walks forward, and takes a snapshot only at requested indexes. Node's Hash.copy() makes those snapshots possible without restarting from byte zero.

const hash = createHash('sha256');
let nextTarget = 0;

for (let i = 0; i < messages.length; i++) {
  hash.update(stableSerialize(messages[i]));
  if (i === targetIndexes[nextTarget]) {
    keys.set(i, hash.copy().digest('hex'));
    nextTarget++;
  }
}

Production code also includes separators, versioning, input validation, and the exact historical byte format. Those details matter: a faster key that differs from the old key would silently invalidate stored state.

Proving compatibility, not merely speed

We kept the old implementation as a test oracle and compared sampled keys from both algorithms. The new path had to be byte-for-byte equivalent at the same prefix boundaries. Tests covered empty content, mixed block types, Unicode, repeated messages, and sparse target indexes.

On a 900-prefix stress fixture, the linear implementation produced the required keys in about 50 ms. The important result was not the benchmark alone; it was the combination of bounded work and identical outputs.

Why the fix worked

The broader lesson is that event-loop stalls are often complexity bugs rather than “Node being slow.” Before adding workers or machines, count how many times the same bytes are parsed, copied, serialized, or hashed.

Operational lessons from Mythos V3

1. Instrument stage boundaries before collecting more logs.
2. Keep observability metadata-only when payloads may be sensitive.
3. Treat old output as an oracle during performance rewrites.
4. Include adversarial large-history fixtures in regression tests.
5. Watch event-loop delay alongside wall time; it distinguishes local synchronous work from waiting on I/O.

Related engineering notes cover preserving reasoning blocks through a relay and avoiding tool-use ID collisions. Together they illustrate the less glamorous part of compatibility engineering: exact bytes, stable identities, and bounded algorithms matter as much as the model call itself.

This article is based on a real Mythos V3 incident. Customer identity, infrastructure addresses, provider details, credentials, and internal topology have been removed.