camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoran Regvart <zo...@regvart.com>
Subject Re: Camel processes only the first 150 rows in CSV
Date Thu, 18 May 2017 07:40:29 GMT
Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.

Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.

The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.

zoran

zoran

On Thu, May 18, 2017 at 8:58 AM, Ambika Das <dasap@outlook.com> wrote:
> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate
the message. I have developed it as a Spring boot application. The problem is when I transfer
a file it takes only the first 150 record and processes. The file is still there in the folder.
If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad



-- 
Zoran Regvart

Mime
View raw message