incubator-kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: kafka.common.OffsetOutOfRangeException
Date Tue, 09 Aug 2011 17:57:16 GMT
Evan,

These exceptions are normal. Basically, the first time a ZK consumer starts
up and finds no previous offset information in zookeeper, it defaults to
Long.MAX_VALUE. This is done to trigger the OffsetOutOfRangeException
handling code path that resets the offset to the correct value based on
"autooffset.reset".

Are you able to produce and consume the data correctly, in spite of these
errors in the logs ?

Thanks,
Neha


On Tue, Aug 9, 2011 at 10:37 AM, Evan Chan <ev@ooyala.com> wrote:

> Hi guys,
>
> I'm following along the Quick Start and ran into an issue when trying to
> use
> a custom Encoder.    I'm basically trying to use a Thrift-encoded message.
>  Here is my setup:
> - Local ZK, Kafka, consumer and producer all on one machine
> - One topic set up with 10 partitions
> - Just one consumer which takes all 10 partitions
> - Consumer has     props.put("autooffset.reset", "largest")
>  in order not to read earlier messages
> - Writing producer and consumer in Scala
>
> ZK and Kafka are started.  Then I start the consumer, followed by the
> producer.
> Consumer shows these errors:
>
> 11/08/09 09:45:45 INFO consumer.ZookeeperConsumerConnector: Consumer
> player-logs-QoS_eng-dynamic-217.v101.mtv-1312908345470 selected partitions
> :
>
> player_logs:0-0,player_logs:0-1,player_logs:0-2,player_logs:0-3,player_logs:0-4,player_logs:0-5,player_logs:0-6,player_logs:0-7,player_logs:0-8,player_logs:0-9
> 11/08/09 09:45:45 INFO consumer.ZookeeperConsumerConnector: end rebalancing
> consumer player-logs-QoS_eng-dynamic-217.v101.mtv-1312908345470 try #0
> [!] Starting Stream
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 1 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 7 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 0 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 8 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 5 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 6 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 4 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 3 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 9 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: player_logs part: 2 offset: 9223372036854775807 from
> 172.16.100.238:9092
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-1
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-7
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-0
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-8
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-5
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-6
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-4
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-3
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-9
> with
> earliest offset 0
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
> out of range
> 11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-2
> with
> earliest offset 0
> ^
>
>
> Kafka shows these errors:
>
> [2011-08-09 09:45:45,842] ERROR error when processing request
> topic:player_logs, part:3 offset:9223372036854775807 maxSize:307200
> (kafka.server.KafkaRequestHandlers)
> kafka.common.OffsetOutOfRangeException: offset 9223372036854775807 is out
> of
> range
> at kafka.log.Log$.findRange(Log.scala:47)
> at kafka.log.Log.read(Log.scala:223)
> at
>
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
> at
>
> kafka.server.KafkaRequestHandlers$$anonfun$3.apply(KafkaRequestHandlers.scala:115)
> at
>
> kafka.server.KafkaRequestHandlers$$anonfun$3.apply(KafkaRequestHandlers.scala:114)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
> at
>
> kafka.server.KafkaRequestHandlers.handleMultiFetchRequest(KafkaRequestHandlers.scala:114)
> at
>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:43)
> at
>
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:43)
> at kafka.network.Processor.handle(SocketServer.scala:268)
> at kafka.network.Processor.read(SocketServer.scala:291)
> at kafka.network.Processor.run(SocketServer.scala:202)
> at java.lang.Thread.run(Thread.java:680)
>
>
>
> Any ideas?
>
> I've occasionally seen errors like these when testing with the default
> StringEncoder as well.
>
> thanks,
> Evan
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message