Designing queues you can reason about

Queues are easy to add and surprisingly hard to explain once they become part of a real system. The useful unit is rarely a single job. It is the story of the work: what requested it, which attempts ran, and where the result ended up.

Start with an identity

Give every unit of work a stable identifier and carry it through retries. A compact event type keeps the contract explicit:

type WorkEvent = {
  workId: string
  attempt: number
  state: "queued" | "running" | "succeeded" | "failed"
  occurredAt: string
}

The identifier lets an interface group related events without guessing from timestamps or payloads.

Measure pressure, not just volume

Queue depth alone is ambiguous. A hundred jobs may be healthy or alarming depending on arrival and processing rates. A simple approximation of pressure is

$$P = \frac{\lambda_{in}}{\lambda_{out}}$$

where $\lambda_{in}$ is the arrival rate and $\lambda_{out}$ is the completion rate. When $P > 1$ for a sustained period, the backlog grows.

Preserve the trail

Do not overwrite the current status in place and discard the history. Append events, derive the current state, and retain enough context to answer the next debugging question.