kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-4547) Consumer.position returns incorrect results for Kafka 0.10.1.0 client
Date Mon, 16 Jan 2017 05:27:26 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ewen Cheslack-Postava updated KAFKA-4547:
-----------------------------------------
    Priority: Blocker  (was: Major)

Bumping to blocker -- this is pretty basic functionality which appears to be broken across
multiple releases. We can reduce it back to a lower priority if we can't get this included,
but given that this has a patch available we should prioritize getting it into the next release.

> Consumer.position returns incorrect results for Kafka 0.10.1.0 client
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-4547
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4547
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 0.10.1.0, 0.10.0.2, 0.10.1.1
>         Environment: Windows Kafka 0.10.1.0
>            Reporter: Pranav Nakhe
>            Assignee: Vahid Hashemian
>            Priority: Blocker
>              Labels: clients
>             Fix For: 0.10.2.0
>
>         Attachments: issuerep.zip
>
>
> Consider the following code -
> 		KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
> 		List<TopicPartition> listOfPartitions = new ArrayList();
> 		for (int i = 0; i < consumer.partitionsFor("IssueTopic").size(); i++) {
> 			listOfPartitions.add(new TopicPartition("IssueTopic", i));
> 		}
> 		consumer.assign(listOfPartitions);		
> 		consumer.pause(listOfPartitions);
> 		consumer.seekToEnd(listOfPartitions);
> //		consumer.resume(listOfPartitions); -- commented out
> 		for(int i = 0; i < listOfPartitions.size(); i++) {
> 			System.out.println(consumer.position(listOfPartitions.get(i)));
> 		}
> 		
> I have created a topic IssueTopic with 3 partitions with a single replica on my single
node kafka installation (0.10.1.0)
> The behavior noticed for Kafka client 0.10.1.0 as against Kafka client 0.10.0.1
> A) Initially when there are no messages on IssueTopic running the above program returns
> 0.10.1.0                   
> 0                              
> 0                              
> 0           
> 0.10.0.1
> 0
> 0
> 0
> B) Next I send 6 messages and see that the messages have been evenly distributed across
the three partitions. Running the above program now returns 
> 0.10.1.0                   
> 0                              
> 0                              
> 2                              
> 0.10.0.1
> 2
> 2
> 2
> Clearly there is a difference in behavior for the 2 clients.
> Now after seekToEnd call if I make a call to resume (uncomment the resume call in code
above) then the behavior is
> 0.10.1.0                   
> 2                              
> 2                              
> 2                              
> 0.10.0.1
> 2
> 2
> 2
> This is an issue I came across when using the spark kafka integration for 0.10. When
I use kafka 0.10.1.0 I started seeing this issue. I had raised a pull request to resolve that
issue [SPARK-18779] but when looking at the kafka client implementation/documentation now
it seems the issue is with kafka and not with spark. There does not seem to be any documentation
which specifies/implies that we need to call resume after seekToEnd for position to return
the correct value. Also there is a clear difference in the behavior in the two kafka client
implementations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message