camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Large file processing with Apache Camel
Date Thu, 28 Feb 2013 13:17:54 GMT
On Thu, Feb 28, 2013 at 1:03 PM, cristisor <cristisor_ac@yahoo.com> wrote:
> After digging more into my problem I found that the slow db access was the
> main issue, maybe you heard before of setting
> sendStringParametersAsUnicode=false in the jdbc driver to dramatically
> increase the performance.
>
> Since the last time I posted here I learned a lot about apache camel and I
> implemented some nice changes, but I'm confronting with a synchronization
> problem right now. I split each line that comes from the file and I enable
> parallel processing:
> from().split(body(String.class).tokenize("\n")).parallelProcessing().process(processor).to()
>
> Inside this processor I take the line, send it to the xml mapper as an input
> stream and in the end I set the xml as the body of the out message. The
> processor's process method is not synchronized and I run into situations
> where the input line has a code "1234" and the output doesn't contain the
> code "1234" but "1235", but this code belongs to another line 100% sure. So
> I should control the synchronization process by myself, right?
>
>

If you run in parallel processing mode, then the lines can be
processed out of order and concurrently.
Also your custom processor will be invoked concurrently by multiple
threads as they process the lines in parallel. So you have to write
your code in a thread safe manner, if you want to access shared
resources.




>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Large-file-processing-with-Apache-Camel-tp5727977p5728292.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Mime
View raw message