camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: File and async/<camel:threads> woes
Date Thu, 06 Sep 2012 09:30:29 GMT
Hi

Why do it so complicated with 2 thread pools. The splitter can just
refer to a custom thread pool profile / thread pool which you can
customize as you want.

Also the fine consumer with preMove will move the file asap, it starts routing.


On Thu, Aug 30, 2012 at 5:24 PM, Ralf Steppacher
<ralf.steppacher@derivativepartners.com> wrote:
> Hallo all,
>
> I have several issues defining an async route in Spring XML.
> Can someone shed some light on what the expected behavior is and what
> would be a bug and/or a misunderstanding on my side?
>
> The stripped down route:
>
> <endpoint id="fileBufferFrom"
> uri="file://${my.dir}/?delete=true&amp;idempotent=true&amp;preMove=inprogress"
/>
>
> <camel:route id="processFromFileBuffer">
>   <camel:from ref="fileBufferFrom" />
>   <camel:threads poolSize="5" maxPoolSize="5" maxQueueSize="1"
> threadName="file consumer" rejectedPolicy="Abort">
>     <camel:convertBodyTo type="java.io.InputStream" />
>     <camel:split streaming="true" parallelProcessing="true">
>         <camel:tokenize token="\r\n" />
>         <camel:to ref="mq.csv" />
>     </camel:split>
>   </camel:threads>
> </camel:route>
>
>
> Camel 2.9.2
> If maxQueueSize > 0 then maxPoolSize + maxQueueSize files are moved from
> the in-folder to the inprogress-folder. If there are more files they are
> moved to the inprogress-folder as soon as the caller thread is free to
> move them and worker threads are available.
> Setting rejectedPolicy="Abort" or callerRunsWhenRejected="false" seems
> to get ignored and the reject policy "CallerRuns" is applied. The source
> is a file endpoint, so I assume the thread with the scanned directory as
> its name is the caller thread? I can see that thread come to live in
> VisualVM.
>
> If maxQueueSize = 0 then all files present in the input directory are
> always moved to the inprogress-folder AND the rejectedPolicy="Abort" is
> honored! There are always maxPoolSize files processed in parallel.
>
>
> Camel 2.10.0
> Independent of the value of maxQueueSize, if rejectPolicy="Abort" then
> the policy is honored, sort of. All files present in the input directory
> are always moved to the inprogress-folder. Only maxPoolSiz +
> maxQueueSize files are being processed. All others stay untouched in the
> "inprogress" folder!
>
> Using <camel:threadPool> and referencing that in <camel:threads> does
> not change the above behavior for 2.9.2 or 2.10.0.
>
>
> The behavior I expected was that with rejectPolicy="Abort" always
> maxPoolSize + maxQueueSize are moved from the in-folder to the
> inprogress-folder and processed from there. As processing of one file
> completes it gets deleted and a file from the in-folder is moved to the
> inprogress-folder on the next poll of the in-folder.
>
>
> Thanks!
> Ralf



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Mime
View raw message