storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladislav Pernin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (STORM-394) Messages has expired, OFFSET_OUT_OF_RANGE, new offset startOffsetTime, no new messages, again and again
Date Tue, 08 Jul 2014 07:33:35 GMT
Vladislav Pernin created STORM-394:
--------------------------------------

             Summary: Messages has expired, OFFSET_OUT_OF_RANGE, new offset startOffsetTime,
no new messages, again and again
                 Key: STORM-394
                 URL: https://issues.apache.org/jira/browse/STORM-394
             Project: Apache Storm (Incubating)
          Issue Type: Bug
    Affects Versions: 0.9.1-incubating
            Reporter: Vladislav Pernin


Issue created here (https://github.com/wurstmeister/storm-kafka-0.8-plus/issues/55) but closed
since the module is maintened under the Storm umbrella now.

I think there might be a case that is not covered :
0) messages in Kafka has expired
1) so offset stored in Zookeeper are no longer valid
2) error OFFSET_OUT_OF_RANGE is thrown
3) getOffset with startOffsetTime
4) retry the fetch with the returned startOffset
5) get an ByteBufferMessageSet but empty

KafkaUtils.fetchMessages seeems to be called again and again with the old offset and we get
to step 2 again.
I guess the new startOffset is not commited to Zookeeper since we do not have new messages.

This can happen in the case of a topology restart, so it goes through the TridentKafkaEmitter.reEmitPartitionBatch



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message