camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Jackson <foamd...@gmail.com>
Subject Re: Scanning/Splitting Large input files
Date Mon, 25 Jan 2010 15:22:34 GMT
Hi,

> The JDK offers a java.util.Scanner which allows you to split a stream
> on-the-fly. Camel leverages this scanner under the covers as well.
>
> For example suppose you want to split a 700mb file pr line then you
> can use the Camel splitter and have it tokenized using \n, which
> should leverage that Scanner under the covers. You can also enable the
> streaming mode of the Splitter which should prevent reading the 700mb
> into memory.
>
> So by enabling streaming and having the big message split by the
> Scanner should allow you to do this with low memory usage.
>
>
> Its the createIterator method on ObjectHelper which the Camel splitter
> will use, if you use the body().tokenize("\n") as the split
> expression.

And this is the case when you use the POJO splitting method?  I
assumed that this was the case as it made the most sense so I have
followed the example in the SplitterPOJOTest

For the Java DSL :
from("direct:start").split().method("mySplitter",
"splitBody").streaming().to("mock:result");

The equivalent spring xml :
<route>
  <from uri="direct:start"/>
  <split streaming="true">
    <bean ref="mySplitter" method="splitBody"/>
    <to uri="mock:result"/>
  </split>
</route>

Given that I cannot tokenize on a simple \n

Thanks,
Kev

Mime
View raw message