camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Hoffer <dhoff...@gmail.com>
Subject Re: Camel ThreadPool maxQueueSize question
Date Fri, 20 Nov 2015 22:09:28 GMT
This part I'm not clear on and it raises more questions.

When using the JDK one generally uses the Executors factory methods to
create either a Fixed, Single or Cached thread tool.  These will use a
SynchronousQueue for Cached pools and LinkedBlockingQueue for Fixed or
Single pools.  In the case of SynchronousQueue there is no size...it simply
hands the new request off to either a thread in the pool or it creates a
new one.  And in the case of LinkedBlockingQueue it uses an unbounded queue
size.  Now it is possible to create a hybrid, e.g. LinkedBlockingQueue with
a max size but its not part of the factory methods or common.  Another
option is the ArrayBlockingQueue which does use a max size but none of the
factory methods use this type.

So what type of thread pool does Camel create for the default thread pool?
Since its not fixed size I assumed it would use SynchronousQueue and not
have a separate worker queue.  However if Camel is creating a hybrid using
a LinkedBlockingQueue or ArrayBlockingQueue is there a way I can change
that to be a SynchronousQueue so no queue?  Or is there a compelling reason
to use LinkedBlockingQueue in a cached pool?

Now this gets to the problem I am trying to solve.  We have a Camel app
that deals with files, lots of them...e.g. all the routes deal with files.
It starts with an sftp URL that gets files off a remote server and then
does a lot of subsequent file processing.  The problem is that if the SFTP
server has 55 files (example) and I start the Camel app it processes them
fine until about 14 or 15 files are left and then it just stops.  The
thread that does the polling of the server stops (at least it appears to
have stopped) and the processing of the 55 files stops, e.g. it does not
continue to process all of the original 55 files, it stops with 14-15 left
to process (and it never picks them up again on the next poll).  And I have
a breakpoint on my custom SftpChangedExclusiveReadLockStrategy and it never
is called again.

Now getting back to the default thread pool and changing it I would like to
change it so it uses more threads and no worker queue (like a standard
Executors cached thread pool) but I'm not certain that would even help as
in the debugger & thread dumps I see that it looks like the SFTP endpoint
uses a Scheduled Thread Pool instead which makes sense since its a polling
(every 60 seconds in my case) operation.  So is there another default pool
that I can configure for Camel's scheduled threads?

All that being said why would the SFTP endpoint just quit?  I don't see any
blocked threads and no deadlock.  I'm new to Camel and just don't know
where to look for possible causes of this.

Thanks,
-Dave


On Thu, Nov 19, 2015 at 11:40 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:

> Yes its part of JDK as it specifies the size of the worker queue, of
> the thread pool (ThreadPoolExecutor)
>
> For more docs see
> http://camel.apache.org/threading-model.html
>
> Or the Camel in Action books
>
>
> On Fri, Nov 20, 2015 at 12:22 AM, David Hoffer <dhoffer6@gmail.com> wrote:
> > I'm trying to understand the default Camel Thread Pool and how the
> > maxQueueSize is used, or more precisely what's it for?
> >
> > I can't find any documentation on what this really is or how it's used.
> I
> > understand all the other parameters as they match what I'd expect from
> the
> > JDK...poolSize is the minimum threads to keep in the pool for new tasks
> and
> > maxPoolSize is the maximum number of the same.
> >
> > So how does maxQueueSize fit into this?  This isn't part of the JDK
> thread
> > pool so I don't know how Camel uses this.
> >
> > The context of my question is that we have a from sftp route that seems
> to
> > be getting thread starved.  E.g. the thread that polls the sftp
> connection
> > is slowing/stopping at times when it is busy processing other files that
> > were previously downloaded.
> >
> > We are using the default camel thread pool that I see has only a max of
> 20
> > threads yet a maxQueueSize of 1000.  That doesn't make any sense to me
> > yet.  I would think one would want a much larger pool of threads (as we
> are
> > processing lots of files) but no queue at all...but not sure on that as I
> > don't understand how the queue is used.
> >
> > -Dave
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message