Engineering Note
The wrong message queue doesn't fail on Day 1. It becomes expensive when you scale.
We’ve used different messaging approaches in Rails systems.
And the biggest lesson was:
RabbitMQ, SQS and Kafka solve different problems.
For background jobs and simple asynchronous processing:
SQS is often enough.
Need reliable delivery without managing another messaging cluster?
SQS keeps the operational burden low.
Need more control over routing and consumers?
RabbitMQ becomes interesting.
Queues, exchanges, routing rules and acknowledgements give you more flexibility.
But when the requirement becomes:
“We need to process a continuous stream of events and replay them later.”
That's where Kafka starts making more sense.
High-throughput event streams.
Multiple consumers.
Event retention.
Replay.
The mistake is choosing Kafka because the system is “large”.
Or choosing SQS because it's “simple”.
The better question is:
What does the system actually need?
* Background jobs → SQS
* Complex message routing → RabbitMQ
* High-volume event streaming and replay → Kafka
And there is another trade-off people underestimate:
Operational complexity.
The more control you get, the more infrastructure you have to operate.
Important learning:
Don't choose a messaging system based on future scale.
Choose it based on the communication pattern you have today, while keeping a clear migration path for tomorrow.
Architecture isn't about picking the most powerful tool.
It's about picking the simplest tool that solves the actual problem.
And the biggest lesson was:
RabbitMQ, SQS and Kafka solve different problems.
For background jobs and simple asynchronous processing:
SQS is often enough.
Need reliable delivery without managing another messaging cluster?
SQS keeps the operational burden low.
Need more control over routing and consumers?
RabbitMQ becomes interesting.
Queues, exchanges, routing rules and acknowledgements give you more flexibility.
But when the requirement becomes:
“We need to process a continuous stream of events and replay them later.”
That's where Kafka starts making more sense.
High-throughput event streams.
Multiple consumers.
Event retention.
Replay.
The mistake is choosing Kafka because the system is “large”.
Or choosing SQS because it's “simple”.
The better question is:
What does the system actually need?
* Background jobs → SQS
* Complex message routing → RabbitMQ
* High-volume event streaming and replay → Kafka
And there is another trade-off people underestimate:
Operational complexity.
The more control you get, the more infrastructure you have to operate.
Important learning:
Don't choose a messaging system based on future scale.
Choose it based on the communication pattern you have today, while keeping a clear migration path for tomorrow.
Architecture isn't about picking the most powerful tool.
It's about picking the simplest tool that solves the actual problem.