A Growing Queue Doesn't Mean You Need More Workers
While working on a market data normalization service, I noticed something familiar:
The number of messages waiting in the queue kept increasing.
My first instinct was simple:
Increase the worker count.
More workers should process more messages.
Problem solved.
And to be fair, sometimes it works.
But today I spent time understanding why it works in some situations and why it can make things worse in others.
The service flow looks roughly like this:
Market data arrives through WebSockets
Data is placed into an in-memory queue
Workers consume messages from the queue
Messages are normalized into a standard format
Results are published to RabbitMQ
To understand what is happening, I usually monitor:
Incoming messages/sec
Outgoing messages/sec
Queue length
A growing queue seems like an obvious signal that the system needs more workers.
But today's researching on it made me realize that a growing queue is only a symptom.
It doesn't tell us what the system is actually waiting for.
That led me into concurrency and parallelism.
A simple example helped:
If a machine has 4 CPU cores and 4 independent tasks:
Running them one after another takes 40 seconds.
Running them simultaneously takes about 10 seconds.
That's parallelism.
But concurrency is different.
Concurrency is about allowing multiple pieces of work to make progress, especially when some of them spend time waiting.
And that distinction turns out to matter a lot.
Consider this situation:
Incoming = 100,000 messages/sec
Outgoing = 80,000 messages/sec
Queue growing
CPU usage = 15%
At first, adding more workers sounds reasonable.
The CPU is mostly idle.
The system is likely waiting on something:
Network operations
RabbitMQ publishing
Lock contention
Other external dependencies
In this case, increasing concurrency may improve throughput because there is unused CPU capacity available.
Now look at a different scenario:
Incoming = 100,000 messages/sec
Outgoing = 80,000 messages/sec
Queue growing
CPU usage = 95%
The queue is still growing.
But the bottleneck is completely different.
The workers are no longer waiting.
They are already busy doing work.
Adding more workers doesn't create more CPU resources.
Instead, the operating system spends more time scheduling threads and performing context switches.
In some cases, throughput can actually get worse.
The biggest realization from today:
A queue tells us that work is arriving faster than it is leaving.
It does not tell us why.
The same symptom can be caused by:
CPU saturation
Network delays
Message broker bottlenecks
Lock contention
Concurrency limits
Other resource constraints
Before optimizing a system, the first question shouldn't be:
How do I make it faster?
The first question should be:
What is the system waiting for?
That single question changes how I think about performance problems.