kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: InvalidMessageSizeException
Date Tue, 12 Feb 2013 05:46:46 GMT
Another way is to figure out a valid offset close to the current offset and
reset the offset in ZK. You can use the tool DumpLogSegment to print out
valid offsets in a log file.

0.6 is pretty old though. I recommend that you upgrade to 0.7.

Thanks,

Jun

On Mon, Feb 11, 2013 at 9:31 AM, Manish Khettry <manish@ooyala.com> wrote:

> One of our consumers keeps getting an invalid message size exception. I'm
> pretty sure that we don't have a message size this big (1.7G). We have two
> other consumer groups consuming messages from the same Kafka instance
> happily over the last few days.
>
> Since we keep the logs around for a fixed interval and this consumer group
> has fallen pretty far behind, is it possible that somehow log truncation is
> causing this? We are on kafka 0.6 BTW.
>
> At this point, I'm inclined to wipe out the "/consumers/<group>/offsets"
> node in zookeeper to get this system going again. Would that be the
> preferred way of getting out of this bad state?
>
> Let me know if there is any other trouble shooting/diagnostics I can run on
>  the system before I reboot!
>
> Manish
>
> [$DATE] ERROR k.c.FetcherRunnable [] - error in FetcherRunnable
> kafka.common.InvalidMessageSizeException: invalid message size:1852339316
> only received bytes:307196 at 0 possible causes (1) a single message larger
> than the fetch size; (2) log corruption
> at
>
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:75)
> ~[rookery-vacuum.jar:na]
> at
>
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:61)
> ~[rookery-vacuum.jar:na]
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:58)
> ~[rookery-vacuum.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:50)
> ~[rookery-vacuum.jar:na]
> at
>
> kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:49)
> ~[rookery-vacuum.jar:na]
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:70)
> ~[rookery-vacuum.jar:na]
> at
>
> kafka.consumer.FetcherRunnable$$anonfun$run$4.apply(FetcherRunnable.scala:80)
> ~[rookery-vacuum.jar:na]
> at kafka.con...
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message