<aside> 💡 Generated by BigBrain. Repeated material was merged while distinct trade-offs, caveats, examples, and operational details were retained.

</aside>

08 - Messaging, Kafka, and Event-Driven Systems

Fundamentals of Messaging Systems

Message Queues

A message queue is a data structure that stores messages in a First-In-First-Out (FIFO) order. Producers add messages to the back of the queue, while consumers retrieve them from the front. This enables asynchronous communication between producers and consumers, decoupling their operations.

Key use cases for message queues include:

• Scheduling and managing background tasks.
• Distributing tasks across multiple worker instances using the competing consumer pattern.
• Implementing retries for failed operations, such as payment processing.
• Buffering data to handle spikes in traffic.

For example, in a file processing system:

• A user uploads a file, and the web server creates a job.
• The job is added to a queue, and the file is stored in object storage.
• A worker process later retrieves the job from the queue and processes the file.

Message Brokers

A message broker is middleware that manages one or more message queues and provides additional features:

• Message Routing: Directs messages to specific queues based on routing keys or criteria.
• Message Transformation: Modifies message content or format.
• Protocol Translation: Converts messages between protocols (e.g., HTTP to JMS).

Message brokers enable the publish-subscribe (pub/sub) pattern, where producers publish messages to topics, and multiple consumers subscribe to those topics. For example: