camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Camel Large File Processing Issues
Date Mon, 18 Apr 2016 07:14:37 GMT
Use streamDownload with localWorkDirectory to store it to a temporary file.

eg read the docs
http://camel.apache.org/ftp2

On Mon, Apr 18, 2016 at 9:03 AM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
> Use streamDownload for the FTP or maybe split up into 2 routes
>
> from ftp -> file
> from file -> split
>
> And if using 2 routes then use done file name or write using temp file
> name so the file consumer only pickup the file when its fully
> downloaded from ftp.
>
> On Mon, Apr 18, 2016 at 8:36 AM, h0mer <mad4comp@gmail.com> wrote:
>> That's what I figured.  But the body is already a String type, so if I do the
>> route above without the marshal(), I don't get the content of the file in
>> the Exchange, but rather I get just the file name and some other content
>> (see the last part of my original post above).
>>
>>
>> Here is my Route without any marshalling or casting the body to a String
>> class.
>>
>>     public void configure() throws Exception {
>>         from(ftpToKafkaObj.getFtpEndpoint())
>>                 .split(body().tokenize("\n"))
>>                 .streaming()
>>                 .process(new FtpToLocalFileProcessor())
>>                 .routeId("ROUTEID::::: " + ftpToKafkaObj.getRouteId())
>>                 .aggregate(constant(true),batchAggregationStrategy())
>>                 .completionPredicate(batchSizePredicate())
>>                 .to(ftpToKafkaObj.getKafkaEndpoint())
>>                 .end();
>>     }
>>
>> This works (as in it goes through without any OOME), but gives me the
>> following in the exchange:
>>
>> <http://camel.465427.n5.nabble.com/file/n5781239/debug_img.png>
>>
>>
>> How do I get the actual String contents line by line, without doing a
>> marshal or casting the body to a String.class type?  The file uses the
>> default UTF-8 charset and is of type string as it is only an apache log
>> file.
>>
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Camel-Large-File-Processing-Issues-tp5781221p5781239.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Mime
View raw message