Now the question arises… How will you introduce this delay? To put it simply, consumer of delay topic (pattern: delay-minutes-x, where x is any number) reads messages from delay topic, then introduce a delay of x minutes, then it publishes the message back to the application topic. Non Blocking Retry Mechanism Delay queue logicĭelay queue comprises of two elements delay topic and its consumer. Application topic is required by the consumer of delay topic to route the message back to the application topic. The reason for that is, since every time the message comes back from the delay topic to the application topic it will be propagated to all its consumer groups and we don’t need to reprocess the message in case some other consumer group failed it. If failed consumer group id extracted from the metadata is equal to the consumer group id of the message then only we should reprocess the message otherwise we should discard the message. Now let’s assume the message comes back to the application topic. If the retry count extracted from the metadata is greater than zero, then we should retry message processing otherwise we should just dump the message to a permanent failure topic since the number of retries for this message exhausted. To better understand why metadata is required let’s assume the message failed. This will help us in identifying whether we should retry the message or not. Note: Add metadata like retry count, application topic and failed consumer group id to the message. The consumer of the delay topic will publish the message back to the application topic (the topic whose message failed) after a certain delay. What can we do to improve this naive implementation? Non blocking retry logicĪ better approach would be to push the failed message to a delay topic if the number of retries is not exhausted and continuing your message consumption. Delay introduced will allow the application to heal and recover from its failure state. So a simple retry logic would be to wait for some time in catch block and reprocess that message. In Kafka you can not skip a message and come back later. Let’s see how we can implement retry mechanism in Kafka! Strategy and Implementation Simple retry logic In the case of an unrecoverable failure you can just dump the message to a permanent failure topic which can then be consumed to report such errors. If it is a recoverable failure you would want a retry mechanism to reprocess that message. It might happen that your message processing fails due to unavailability or errors in the dependent system. Kafka provides consumer apis to poll messages from the broker that can be processed accordingly. Use Case: Retry message consumption in the case of processing failure

There are many use cases for delaying message consumption but I would like to stress on the most important one :. However this does not mean you can’t implement it yourself.

The usage of Apache Kafka is growing tremendously because of its unique design and high performance, but it lacks the support for delay queues and dead letter queues. RabbitMQ has a plugin for this while ActiveMQ has this feature built into the message broker. Sometimes you want to delay the delivery of your messages so that subscribers don’t see them immediately.

Delayed job enqueue