Queueing theory in plain English
Queues in knowledge work include ticket backlogs, pull requests awaiting review, decisions awaiting a meeting, features needing a specialist, and incidents pending diagnosis. Partly finished work waiting for its next step is a queue too.
Matching average rates hide the timing of work. Four items arriving together leave three waiting if only one can be served at a time, even when arrivals average one per hour and capacity is one per hour. Variation in item size, interruptions, rework, reviews, and handoffs add enough waiting to dominate elapsed time.
Core quantities
- Arrival rate (λ): how fast work enters the system.
- Service rate (μ): how fast the constrained step can finish work.
- Utilization (ρ) = λ / μ: average load at the constraint.
- Variability (Ca, Cs): coefficients of variation for arrivals and service times. C = 0 is deterministic, C = 1 is exponential (Poisson arrivals), C > 1 is bursty or fat-tailed. Ca measures arrival unevenness; Cs measures service-time unevenness.
- WIP (L): average number of items in the system at any moment.
- Lead time (W): average time an item spends in the system, from entry to exit.
Worked example: if λ = 4 items/hour and μ = 5 items/hour, then ρ = 0.80. Under M/M/1, W = 1 / (μ − λ) = 1 hour. Little's Law then gives L = λ × W = 4 items in the system on average.
Little's Law
In a stable system observed over a meaningful interval, average work in the system equals average throughput multiplied by average time in the system. This is Little's Law.
| Variable | Mathematical meaning | In high-software-content work | Order-of-magnitude notes |
|---|---|---|---|
| L | Average number of items in the system. | Work in process: ideas, features, fixes, approvals, reviews, blocked work, and partially done delivery items. | Count with one consistent unit and system boundary. |
| λ | Average arrival rate, equal to throughput rate in a stable system. | Rate at which work enters, or exits as completed value once the system is stable. | Use the completed throughput rate across the same boundary used for WIP and lead time. |
| W | Average time an item spends in the system. | Lead time from accepted work to delivered or validated outcome. | Use percentiles alongside averages because product work is usually skewed. |
Little's Law does not say why waiting happens. It says the quantities must balance. If throughput is capped and WIP rises, lead time rises with it.
Why high utilization bites
M/M/1 describes a single server with random arrivals, random service times, and a stable arrival rate below capacity. It is a simplified model rather than a representation of software development. Even here, waiting rises steeply as arrival rate approaches service capacity.
Rewriting μ − λ as μ(1 − ρ) gives W = (1/μ) ÷ (1 − ρ). Average time in the system is average service time divided by the spare-capacity fraction. At 50% utilization, it is two service times. At 90%, it is ten, and at 99%, a hundred. The denominator approaches zero, producing a hyperbola.
Poisson arrivals and exponential service times are assumptions used in the derivation through a birth-death Markov chain. They still constrain the result, even though the final expression contains only the rates. The M/M/1 appendix works through that derivation.
Where variability enters
Kingman's approximation accounts for uneven arrivals and service times. These matter in product work, where rework, blocked reviews, changing priorities, and dependencies alter both the timing and size of tasks.
The formula has three multiplicative factors: the utilization term ρ / (1 − ρ), which grows steeply near saturation; the variability term (Ca2 + Cs2) / 2, which scales with the combined unevenness of arrivals and service; and E[S], the average service time, which sets the baseline wait. Variability amplifies waiting at any utilization, but near saturation there is no slack to absorb it.
Connection to Theory of Constraints
Theory of Constraints (TOC) uses the constraint to guide management action. Queueing theory explains why waiting builds there.
TOC holds that a system's throughput is governed by its constraint. Other decisions should protect that flow, keeping the constraint supplied without letting other steps overload it.
From utilization to flow
Henrik Kniberg's resource utilization trap demonstration puts two release policies alongside each other. Both lanes receive the same bursty arrivals and variable task sizes. One starts work aggressively to keep people busy. The other caps WIP and accepts local slack to absorb variation.
Bursty arrivals create different queues from steady arrivals at the same average load. Fat-tailed item sizes add to the variability at each stage. Starting more work raises WIP and, by Little's Law, lead time. Read throughput, lead time, and WIP together to see how the policies affect flow.
See Kniberg's resource utilization trap video for the original visual demonstration.
Continue exploring
References
- Little, J. D. C. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3), 383–387.
- Kingman, J. F. C. (1961). The single server queue in heavy traffic. Mathematical Proceedings of the Cambridge Philosophical Society, 57(4), 902–904.
- Kleinrock, L. (1975). Queueing Systems, Volume 1: Theory. Wiley.
- Goldratt, E. M., & Cox, J. (1984). The Goal: A Process of Ongoing Improvement. North River Press.
- Goldratt, E. M. (1990). What Is This Thing Called Theory of Constraints and How Should It Be Implemented? North River Press.
- Reinertsen, D. G. (2009). The Principles of Product Development Flow: Second Generation Lean Product Development. Celeritas Publishing.
- Kniberg, H. (c. 2012). Resource Utilization Trap [video]. Resource Utilization Trap on YouTube.