kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip O'Toole <phi...@loggly.com>
Subject Re: High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue
Date Wed, 21 Aug 2013 16:26:42 GMT
The high-level consumer will block if a) there are no more messages
available, b) the next message available is larger than the maximum fetch size you
have specified, or c) your client code simply stops pulling messages from the
iterator (the blocking queue will fill up).

Not sure if this applies to what you are seeing, but you should be aware of
these conditions.

Philip

On Thu, Aug 22, 2013 at 12:10:54AM +0800, Frank Yao wrote:
> Hi,
> 
> My java consumer client will suddenly stop consuming the stream. While I
> enabled the 'TRACE' log, I found that this stop always happened in the end
> of a queuechunk consumption - after several times' chunk actually. What
> weird is that in log there are not any useful information. It occurred
> suddenly.
> 
> I use jstack to figure out what happened and I found that perhaps the
> thread is locked:
>   java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000000fa399970> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51
> 
> After google, I found some body has met this problem before, but no
> solution there.
> http://mail-archives.apache.org/mod_mbox/kafka-users/201211.mbox/%3CCANZjK9i87enoPY15rzh2Bg4D8+H1jvkSCkro=f3EROjn_4T=rA@mail.gmail.com%3E
> 
> If any one can offer any help, I'll really appreciate it.
> 
> -----------------
> 
> Frank Yao
> @VIPSHOP, Shanghai, China

-- 
Philip O'Toole

Senior Developer
Loggly, Inc.
San Francisco, CA.
www.loggly.com

Come join us!
http://loggly.com/company/careers/

Mime
View raw message