kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "joechen8310@gmail.com (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-7318) Should introduce a offset reset policy to consume only the messages after subscribing
Date Tue, 21 Aug 2018 06:08:00 GMT
joechen8310@gmail.com created KAFKA-7318:
--------------------------------------------

             Summary: Should introduce a offset reset policy to consume only the messages
after subscribing
                 Key: KAFKA-7318
                 URL: https://issues.apache.org/jira/browse/KAFKA-7318
             Project: Kafka
          Issue Type: Improvement
          Components: consumer
    Affects Versions: 2.0.0, 1.1.1, 1.1.0
            Reporter: joechen8310@gmail.com


On our situation, we want the consumers only consume the messages which was produced after
subscribing.   

Currently, kafka support 3 policies with auto.offset.reset, but seems both of them can not
support the feature we want.
 * {{latest}} (the default) , if a consumer subscribe a new topic and then close, during
these times, there are some message was produced,  the consumer can not poll these messages.
 * earliest , consumer may consume all the messages on the topic  before subscribing.
 * none, not in this scope.

Before version 1.1.0, we make the consumer poll and commit  after subscribe as below, this
can mark the offset to 0 and works (enable.auto.commit is false) .

 
{code:java}
consumer.subscribe(topics, consumerRebalanceListener);
if(consumer.assignment().isEmpty()) {
   consumer.poll(0);
   consumer.commitSync();
}
{code}
After upgrade the clients to >=1.1.0,  it is broke. Seems it was broke by the fix [KAFKA-6397|https://github.com/apache/kafka/commit/677881afc52485aef94150be50d6258d7a340071#diff-267b7c1e68156c1301c56be63ae41dd0],
but I am not sure about that.  Then I try to invoke the consumer.position(partitions) in onPartitionsAssigned
of ConsumerRebalanceListener,  it works again. but it looks strangely that get the position but
do nothing.  

 

so we want to know whether there is a formal way to do this, maybe introduce another policy
for auto.offset.reset to only consume the message after  the consumer subscribing is perfect.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message