camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: How to determine when one's streamed to the end of a file?
Date Wed, 30 Jun 2010 04:35:07 GMT
Hi

In streaming mode there is currently no way of knowing when the stream is ended.
We could maybe detect this and add a header to the last Exchange so
you can use this fact to know there are no more splits.

I have created a ticket to improve on this
https://issues.apache.org/activemq/browse/CAMEL-2880


On Tue, Jun 29, 2010 at 7:27 AM, garyk70 <gkirrene@gmail.com> wrote:
>
> Good evening,
>
> I'm attempting to stream a large CSV file, split on each row, aggregrate the
> rows in batches, and insert the batched rows into a database table.  My only
> issue is that I can't figure out how to determine when I've reached the end
> of the file (and therefore the end of the aggregations and inserts).  Is
> there a ready means of detecting the end of the file (and if not, perhaps a
> more effective route design than mine)?
>
> Here's my route:
>
> from("file:///Documents/tmp?fileName=TEST.dat")
> .process(new Processor() {
>        public void process(Exchange exchange) throws Exception {
>                dao.truncateTable();
>                dao.disableUniqueKeyIndex();
>        }
> }).split(body().tokenize()).streaming().aggregate(header("CamelFileName"),
> new AggregationStrategy() {
>
>        @SuppressWarnings("unchecked")
>        public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
>
>                if (oldExchange == null) {
>                        oldExchange = newExchange.copy();
>                        oldExchange.getIn().setBody(new ArrayList());
>                }
>
>                List<String> rows = oldExchange.getIn().getBody(List.class);
>                String row = newExchange.getIn().getBody(String.class);
>                rows.add(row);
>
>                return oldExchange;
>        }
> }).completionSize(1000).completionTimeout(30000).process(new Processor() {
>
>        public void process(Exchange exchange) throws Exception {
>
>                List<String> rows = exchange.getIn().getBody(List.class);
>                dao.loadDatFileRows(rows);
>        }
> // read pretend "Status" property
> }).end().filter(property("STATUS").isEqualTo("FINISHED")).process(new
> Processor() {
>
>        public void process(Exchange exchange) throws Exception {
>                dao.rebuildUniqueKeyIndex();
>                dao.updateStatistics();
>        }
> });
>
> Thanks in advance for any guidance.
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-determine-when-one-s-streamed-to-the-end-of-a-file-tp511784p511784.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message