camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen" ...@silverbullet.dk>
Subject RE: Splitter for big files
Date Thu, 04 Sep 2008 04:07:25 GMT
Hi

CAMEL-876 cover any types of files. The batch just means that camel will chop its internal
list of "lines" in a fixed list size, and then process the "lines" as batches, eg:

After
=====
Read 20 lines
Process 20 exchanges
Read 20 lines
Process 20 exchanges
...


Before
======
Read *all* lines
Process *all* lines


Workaround
==========
If you are using ServiceMix already then you are in luck as you have found a solution in it
that works. ServiceMix  has integration to camel with the camel-jbi component. However I am
not familiar with how this works. The user forum at servicemix will be able to help.

However we will soon start working on these issues you have with Camel so we are able to help
you with Camel only. So if you have a little patience and is able to test on-the-fly then
that would be much helpful.



Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Bart Frackiewicz [mailto:bart@open-medium.com] 
Sent: 3. september 2008 23:16
To: camel-user@activemq.apache.org
Subject: Re: Splitter for big files

Hi Claus,

Claus Ibsen schrieb:
> 
> Could you create a JIRA ticket for this improvement?

Thank you for creating the JIRA tickets. Do CAMEL-876 cover the XPATH 
issue? The batch functionality was new for me.

> Btw how big is the files you use? 

We process old datasets up to 1gb, the XML files are about 250mb.

Can i create a workaround for this? A colleague of mine found a good 
implementation in Servicemix, which splits the lines and send the new 
exchanges direct on the message bus. I am not so experienced with Camel 
to measure this. Maybe you can give me a hint how you would solve it.

Bart

Mime
View raw message