kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue
Date Wed, 21 Aug 2013 16:43:01 GMT
Thanks for listing the possible conditions for a stalled consumer, Philip.
I added this to the FAQ -
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F

Frank,

Let us know if the FAQ above helps to understand your issue or can be
improved.

Thanks,
Neha


On Wed, Aug 21, 2013 at 9:26 AM, Philip O'Toole <philip@loggly.com> wrote:

> The high-level consumer will block if a) there are no more messages
> available, b) the next message available is larger than the maximum fetch
> size you
> have specified, or c) your client code simply stops pulling messages from
> the
> iterator (the blocking queue will fill up).
>
> Not sure if this applies to what you are seeing, but you should be aware of
> these conditions.
>
> Philip
>
> On Thu, Aug 22, 2013 at 12:10:54AM +0800, Frank Yao wrote:
> > Hi,
> >
> > My java consumer client will suddenly stop consuming the stream. While I
> > enabled the 'TRACE' log, I found that this stop always happened in the
> end
> > of a queuechunk consumption - after several times' chunk actually. What
> > weird is that in log there are not any useful information. It occurred
> > suddenly.
> >
> > I use jstack to figure out what happened and I found that perhaps the
> > thread is locked:
> >   java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00000000fa399970> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
> > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> > at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51
> >
> > After google, I found some body has met this problem before, but no
> > solution there.
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201211.mbox/%3CCANZjK9i87enoPY15rzh2Bg4D8+H1jvkSCkro=f3EROjn_4T=rA@mail.gmail.com%3E
> >
> > If any one can offer any help, I'll really appreciate it.
> >
> > -----------------
> >
> > Frank Yao
> > @VIPSHOP, Shanghai, China
>
> --
> Philip O'Toole
>
> Senior Developer
> Loggly, Inc.
> San Francisco, CA.
> www.loggly.com
>
> Come join us!
> http://loggly.com/company/careers/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message