camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Processing large files in Camel
Date Sat, 15 Sep 2012 06:24:36 GMT
Hi

I suggest to take a moment to look more at the docs
http://camel.apache.org/file2

For example on that link there is a red box on top that talks about
the issue of reading files which are currently being written by
another process. There is a number of options you can configure to
chose among. And if that don't suit you, you can implement your own
custom read lock strategy.

And for sending the "message" to AMQ you would need to do a message
transformation so you don't send the entire file content with it, but
only the meta-data you want. And in the format you want.

See the EIP for that
http://camel.apache.org/message-translator.html

There is more EIPs here
http://camel.apache.org/enterprise-integration-patterns.html

And since you have to leave the files in place, you would have to
consider at "what to do" in case the server is restarted.
Then the previous "scanned" files may be picked up again. Is this
okay, or do you need to keep "book" of  that.
If so check out the idempotent consumer option on the file component,
as well as the EIP
http://camel.apache.org/idempotent-consumer.html


On Fri, Sep 14, 2012 at 5:03 PM, gramanero <gramanero@gmail.com> wrote:
> I am attempting to understand how the Camel File/File2 component works as I
> have a need to use this type of capability in an upcoming project.
>
> Goal: Write a Camel spring-based route that will detect files in a folder
> structure, wait until the entire file has been written, and place an entry
> onto an ActiveMQ queue that indicates the file name, path, and length, and
> leave the file in place (i.e. do not let Camel move it to a .camel folder).
>
> Camel Route:
>     <route id="newFileReceived" startupOrder="1">
>       <from
> uri="file:E:\NewDataReceived?recursive=true&amp;maxMessagesPerPoll=500&amp;noop=true&amp;delay=60000"/>
>       <to uri="activemq:queue:data.received"/>
>     </route>
>
> Problem: For small files this route works as expected. For large files, I am
> seeing a .camelLock file appear every N seconds (which corresponds to the
> configured 'delay'setting) and then disappear. I am assuming that this is
> Camel trying to get a read lock on the file, but I cannot find any docs on
> this type of behavior. I am also guessing that it keeps appearing and
> disappearing due to the file size?? The size of the files that I am dealing
> with are anywhere from 7Gb and up. This may not be a realistic use case for
> what we will be dealing with, but I want to verify that writing files will
> only ever put an entry onto the queue once the file have been completely
> written and not before.
>
> Camel version: Whatever is baked into apache-servicemix-4.4.1-fuse-03-06
>
> Any suggestions/help is greatly appreciated.
>
> Thank you.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Processing-large-files-in-Camel-tp5719376.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Mime
View raw message