storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Landman <>
Subject KafkaSpout offsets
Date Thu, 24 Jul 2014 19:27:07 GMT
In nathanmarz/storm-contrib project there was a KafkaConfig that had a
forceOffsetTime.  In our code someone had documented that calling this with
different values would affect the offsets in the following way:

-2 Will start at the beginning (earliest message) of the topic
-1 Will start at the end (latest message) of the topic
-3 Will start where the spout left off
And anthing >0 will start at the specified offset.

In the new project external/storm-kafka there is also a KafkaConfig and I
see that it exposes
public boolean forceFromStart = false;
public long startOffsetTime = kafka.api.OffsetRequest.EarliestTime();
public long maxOffsetBehind = 100000;
public boolean useStartOffsetTimeIfOffsetOutOfRange = true;

By default does this mean the spout will start at the beginning of the
topic?  What does the forceFromStart do?  If we want to start from whatever
offset the spout was last processing, is there anyway to do this?

View raw message