camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Ilyin <alexande...@madadsmedia.com>
Subject RecipientList thread pool problem
Date Mon, 20 Apr 2015 13:19:39 GMT
Hi,

I have a problem with RecipientList pattern. I'm using it in a parallel
processing mode which means it is backed up by a thread pool. The
configuration is as follows:

<threadPool id="pool" threadName="My RecipientList" poolSize="10"
maxPoolSize="200" />

<recipientList id="bidderResponseGatherer"
strategyRef="gatherBidderResponses" parallelProcessing="true"
stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true"
executorServiceRef="pool" >
    <header>RecipientsList</header>
</recipientList>

The problem is that pool threads aren't released properly sometimes. After
a few hours following the start of an application I can see using jstack
tool that some threads are stuck in a WAITING state:

"Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10
tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fb4dc319940> (a
java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
        at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
        at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327)
        at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)
        at
org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153)
        at
org.apache.camel.processor.RecipientList.process(RecipientList.java:112)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
        at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
        at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at
org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136)
        at
org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108)
        at
org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199)
        at
org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103)
        at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at
org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Number of such threads grows gradually until the application isn't able to
respond to the requests. I've found very similar ticket in your Jira (
https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there
that the problem is in the 3rd party library mqtt-client. However, I'm not
using this library, I'm only using Netty to make http requests but still
see a very similar behaviour.

I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as
suggested here
http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html.
But this didn't help too. Also I've noticed that thread is becoming
unavailable after the timeout specified for the calls has been reached.
Although not every timeout leads to a thread leak.

Can you suggest what else can be checked? Am I missing something? Or may be
this is a bug in Camel?


-- 
 Alexander Ilyin
*tel: 856-874-8928 <856-874-8928>*  Director of Engineering
alexander.i@madadsmedia.com  717 Fellowship Rd. Suite D
Mount Laurel, NJ 08054

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message