camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Ilyin <alexande...@madadsmedia.com>
Subject Re: RecipientList thread pool problem
Date Wed, 29 Apr 2015 15:01:41 GMT
Hi Claus,

thank you for your answer.
I hadn't configured netty request timeout before I wrote my first letter.
Now I've read that tasks may keep running when using RecipientList timeout
and have set up netty request timeout instead of RecipientList timeout as
you suggested. But the situation didn't change much. NettyOrderedWorker
threads are still getting stuck in a WAITING state after some time
following the start of an app as well as RecipientList threads.
What else can be checked? I can provide additional info if needed.

On Mon, Apr 20, 2015 at 4:19 PM, Alexander Ilyin <
alexander.i@madadsmedia.com> wrote:

> Hi,
>
> I have a problem with RecipientList pattern. I'm using it in a parallel
> processing mode which means it is backed up by a thread pool. The
> configuration is as follows:
>
> <threadPool id="pool" threadName="My RecipientList" poolSize="10"
> maxPoolSize="200" />
>
> <recipientList id="bidderResponseGatherer"
> strategyRef="gatherBidderResponses" parallelProcessing="true"
> stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true"
> executorServiceRef="pool" >
>     <header>RecipientsList</header>
> </recipientList>
>
> The problem is that pool threads aren't released properly sometimes. After
> a few hours following the start of an application I can see using jstack
> tool that some threads are stuck in a WAITING state:
>
> "Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10
> tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00007fb4dc319940> (a
> java.util.concurrent.CountDownLatch$Sync)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>         at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
>         at
> org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327)
>         at
> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)
>         at
> org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153)
>         at
> org.apache.camel.processor.RecipientList.process(RecipientList.java:112)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
>         at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
>         at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
>         at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
>         at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>         at
> org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136)
>         at
> org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108)
>         at
> org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199)
>         at
> org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>         at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
>         at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
>         at
> org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> Number of such threads grows gradually until the application isn't able to
> respond to the requests. I've found very similar ticket in your Jira (
> https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there
> that the problem is in the 3rd party library mqtt-client. However, I'm not
> using this library, I'm only using Netty to make http requests but still
> see a very similar behaviour.
>
> I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as
> suggested here
> http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html.
> But this didn't help too. Also I've noticed that thread is becoming
> unavailable after the timeout specified for the calls has been reached.
> Although not every timeout leads to a thread leak.
>
> Can you suggest what else can be checked? Am I missing something? Or may
> be this is a bug in Camel?
>
>
> --
>  Alexander Ilyin
> *tel: 856-874-8928 <856-874-8928>*  Director of Engineering
> alexander.i@madadsmedia.com  717 Fellowship Rd. Suite D
> Mount Laurel, NJ 08054
>



-- 
 Alexander Ilyin
*tel: 856-874-8928 <856-874-8928>*  Director of Engineering
alexander.i@madadsmedia.com  717 Fellowship Rd. Suite D
Mount Laurel, NJ 08054

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message