Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Date: Thu, 23 Jun 2016 17:33:16 +0000 (UTC)
From: "Krish Iyer (JIRA)" <jira@apache.org>
To: dev@kafka.apache.org
Message-ID: <JIRA.12982404.1466703176000.12046.1466703196173@Atlassian.JIRA>
In-Reply-To: <JIRA.12982404.1466703176000@Atlassian.JIRA>
References: <JIRA.12982404.1466703176000@Atlassian.JIRA> <JIRA.12982404.1466703176179@arcas>
Subject: [jira] [Created] (KAFKA-3895) Implement a default queue for expired
 messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 23 Jun 2016 17:33:23 -0000

Krish Iyer created KAFKA-3895:
---------------------------------

             Summary: Implement a default queue for expired messages
                 Key: KAFKA-3895
                 URL: https://issues.apache.org/jira/browse/KAFKA-3895
             Project: Kafka
          Issue Type: New Feature
          Components: core
            Reporter: Krish Iyer
            Priority: Minor


As discussed in the mailing list, kafka currently does not support a dead letter queue-like feature where all the expired messages can be sent to.

A high-level design of such a feature can be as follows:
When kafka needs to expire a message from topic 'X', it sends the message to 'X-expired' topic, rather than deleting it. If 'X' is 'X-expired', dequeue and requeue the message in the expired topic.
Given that this topic can receive duplicate messages (same expired message multiple times), the consumer of this topic needs to ensure duplicate message handling.

Open question:
Does the message need a timestamp to figure out when did it move from a normal queue to a dead letter queue. Does it need a flag to specify whether it had expired from the X-expired topic itself?


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)