Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAA8BW8F0RvSRd5EghruEApuBQ2uTF=-e-xkzvv=wvmqOD3qGsQ@mail.gmail.com>
References: 
 <CAA8BW8HLtMt-8rS78En-QAUCArVQxXjYHqbGmwp6XiUOMWHJrQ@mail.gmail.com>
 <CAGB5yNnLxYh3n2ToMm3sSt+G7CC-XnWsniPnVh=8BWi2SRroLQ@mail.gmail.com>
 <CAA8BW8F0RvSRd5EghruEApuBQ2uTF=-e-xkzvv=wvmqOD3qGsQ@mail.gmail.com>
From: Claus Ibsen <claus.ibsen@gmail.com>
Date: Sat, 21 Nov 2015 09:40:55 +0100
Message-ID: 
 <CAGB5yNndvcYgT-Qq-E5e8JTj_dwK8ADydSn+==gD1ZNc+Yif6w@mail.gmail.com>
Subject: Re: Camel ThreadPool maxQueueSize question
To: "users@camel.apache.org" <users@camel.apache.org>
Content-Type: text/plain; charset=UTF-8

Set queue size to 0 or -1 or to have no queue and a direct handover
with that synchronous queue. And then configure the rejected policy to
decide what to do if there is no free threads, such as reject or use
current thread etc.

On Fri, Nov 20, 2015 at 11:09 PM, David Hoffer <dhoffer6@gmail.com> wrote:
> This part I'm not clear on and it raises more questions.
>
> When using the JDK one generally uses the Executors factory methods to
> create either a Fixed, Single or Cached thread tool.  These will use a
> SynchronousQueue for Cached pools and LinkedBlockingQueue for Fixed or
> Single pools.  In the case of SynchronousQueue there is no size...it simply
> hands the new request off to either a thread in the pool or it creates a
> new one.  And in the case of LinkedBlockingQueue it uses an unbounded queue
> size.  Now it is possible to create a hybrid, e.g. LinkedBlockingQueue with
> a max size but its not part of the factory methods or common.  Another
> option is the ArrayBlockingQueue which does use a max size but none of the
> factory methods use this type.
>
> So what type of thread pool does Camel create for the default thread pool?
> Since its not fixed size I assumed it would use SynchronousQueue and not
> have a separate worker queue.  However if Camel is creating a hybrid using
> a LinkedBlockingQueue or ArrayBlockingQueue is there a way I can change
> that to be a SynchronousQueue so no queue?  Or is there a compelling reason
> to use LinkedBlockingQueue in a cached pool?
>
> Now this gets to the problem I am trying to solve.  We have a Camel app
> that deals with files, lots of them...e.g. all the routes deal with files.
> It starts with an sftp URL that gets files off a remote server and then
> does a lot of subsequent file processing.  The problem is that if the SFTP
> server has 55 files (example) and I start the Camel app it processes them
> fine until about 14 or 15 files are left and then it just stops.  The
> thread that does the polling of the server stops (at least it appears to
> have stopped) and the processing of the 55 files stops, e.g. it does not
> continue to process all of the original 55 files, it stops with 14-15 left
> to process (and it never picks them up again on the next poll).  And I have
> a breakpoint on my custom SftpChangedExclusiveReadLockStrategy and it never
> is called again.
>
> Now getting back to the default thread pool and changing it I would like to
> change it so it uses more threads and no worker queue (like a standard
> Executors cached thread pool) but I'm not certain that would even help as
> in the debugger & thread dumps I see that it looks like the SFTP endpoint
> uses a Scheduled Thread Pool instead which makes sense since its a polling
> (every 60 seconds in my case) operation.  So is there another default pool
> that I can configure for Camel's scheduled threads?
>
> All that being said why would the SFTP endpoint just quit?  I don't see any
> blocked threads and no deadlock.  I'm new to Camel and just don't know
> where to look for possible causes of this.
>
> Thanks,
> -Dave
>
>
> On Thu, Nov 19, 2015 at 11:40 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>
>> Yes its part of JDK as it specifies the size of the worker queue, of
>> the thread pool (ThreadPoolExecutor)
>>
>> For more docs see
>> http://camel.apache.org/threading-model.html
>>
>> Or the Camel in Action books
>>
>>
>> On Fri, Nov 20, 2015 at 12:22 AM, David Hoffer <dhoffer6@gmail.com> wrote:
>> > I'm trying to understand the default Camel Thread Pool and how the
>> > maxQueueSize is used, or more precisely what's it for?
>> >
>> > I can't find any documentation on what this really is or how it's used.
>> I
>> > understand all the other parameters as they match what I'd expect from
>> the
>> > JDK...poolSize is the minimum threads to keep in the pool for new tasks
>> and
>> > maxPoolSize is the maximum number of the same.
>> >
>> > So how does maxQueueSize fit into this?  This isn't part of the JDK
>> thread
>> > pool so I don't know how Camel uses this.
>> >
>> > The context of my question is that we have a from sftp route that seems
>> to
>> > be getting thread starved.  E.g. the thread that polls the sftp
>> connection
>> > is slowing/stopping at times when it is busy processing other files that
>> > were previously downloaded.
>> >
>> > We are using the default camel thread pool that I see has only a max of
>> 20
>> > threads yet a maxQueueSize of 1000.  That doesn't make any sense to me
>> > yet.  I would think one would want a much larger pool of threads (as we
>> are
>> > processing lots of files) but no queue at all...but not sure on that as I
>> > don't understand how the queue is used.
>> >
>> > -Dave
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>


-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2