Message Brokers

Message brokers enable asynchronous communication between services, improve decoupling, and increase system resilience.
However, they also introduce concerns related to message delivery semantics, duplicate processing, and concurrency.

Applications must be designed to handle scenarios where messages may:

• be delivered more than once
• arrive out of order
• be re-queued
• be retried due to processing failures

To achieve correct and safe message processing, developers must design consumers that are idempotent, or use patterns that ensure only one consumer processes a given message.

Competing Consumers Pattern

When multiple instances of the same service read from the same queue or topic partition, the recommended approach to guarantee that exactly one instance processes a given message is the Competing Consumers Pattern.

This pattern ensures:

• multiple consumers can scale horizontally
• only one consumer instance processes each message
• load is distributed across consumer instances
• consumer failures do not cause message loss

Reference:
https://docs.microsoft.com/en-us/azure/architecture/patterns/competing-consumers

RabbitMQ

RabbitMQ implements the Competing Consumers Pattern natively.

When multiple consumers subscribe to the same queue:

• RabbitMQ delivers each message to only one consumer
• Message acknowledgements ensure messages are not lost
• If a consumer fails, unacknowledged messages are re-delivered to another consumer

Because of this built-in behavior, RabbitMQ is a safe and straightforward broker for one-at-a-time message processing.

Requirements When Using RabbitMQ

• Ensure explicit acknowledgements (ack) are enabled.
• Ensure queues are durable and messages are persistent when needed.
• Ensure consumers handle retries and duplicate deliveries safely.
• Prefer idempotent message handlers wherever possible.

Kafka

Kafka is fundamentally different from RabbitMQ.
It delivers messages based on:

• topic partitions
• consumer groups

Consumer Groups and Delivery Guarantees

To ensure that only one instance of an application receives a given message:

• All consumers of the same application must share the same consumer-group-id.
• Kafka will then distribute partitions across consumer instances.
• Each partition is consumed by exactly one consumer in the group at a time.

Important Notes for Kafka

• Kafka does not guarantee exactly-once processing unless additional mechanisms are used.
• Consumers must be designed to handle duplicate messages.
• Keep partition counts aligned with expected consumer scaling.
• When ordering is important, ensure all relevant messages are sent to the same partition.

Summary

To ensure safe and predictable message processing:

• Use the Competing Consumers Pattern for horizontally scaled consumers.
• RabbitMQ naturally enforces single-consumer semantics per message.
• Kafka requires proper consumer-group configuration.
• Always design consumers to be idempotent and robust to retries or duplicates.

Correct message broker usage ensures reliable inter-service communication across DIT systems.