camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Lipp <Christian.L...@xion.at>
Subject AW: IdempotentRepository
Date Mon, 09 Jul 2012 20:42:49 GMT
I think using filter is not possible since I cannot detect the end of the transmission. An
implementation of IdempotentRepository should work because 

- contains() works like filter.accept(): directories and files are handed over but since they
are all strings I have to know the structure to distinguish between them. Anyway I can count
(or even remember) the number of files
- add() only receives the files, not the directories. I know that I am finished when I receive
al files I accepted in contains().

Anyway I am still astound that there is no better solution or that it is not possible to receive
the end of the ftp polling.
Kind regards, Christian

 
________________________________________
Re: AW: IdempotentRepository

For repository contains() is called for directories and files, while add() is called only
for files.
But it is much easier to use filter instead of repository as you recommended, so I switched
to filter.

However the original problem is still there: I mark directories as processed, since these
contain the date-information and I do this when I switch to a new directory: because the input
is sorted I know that the old directory is handled and I mark it as processed.

But when I read the last directory I don't receive a "it is over" message inside the filter,
so the next time the route is executed, it copies the last directory again and I would like
to solve this.

Regards, Christian

-----Urspr√ľngliche Nachricht-----
Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Gesendet: Freitag, 06. Juli 2012 15:04
An: users@camel.apache.org
Betreff: Re: IdempotentRepository

Hi

You can use a filter, and then some custom logic (eg a pojo) and return true|false to accept
the file.
Then you dont need the idempotent pattern.

And from Camel 2.10 onwards the filter is now also invoked for directories. So you can skip
traversing down in directories you do not want to.


On Fri, Jul 6, 2012 at 8:17 AM, Christian Lipp <Christian.Lipp@xion.at> wrote:
> Hello!
>
> I am fetching files from a FTP server (severall GB for the next
> years). The files are produced daily in directories which correspond
> to the date, like
>
> -       20120501
> -       20120502
> -       ...
>
> I have only read rights and I am not the only consumer. This means that they keep the
last month or so on the server and I fetch on a daily base. To avoid that I am fetching files
twice I want to use an IdempotentRepository implementation. I don't want to save each file
in a database or in a text file because the service will run for years and this is just unnecessary
data.
>
> What I want to store is the last processed date only. This handles just the directories
and would mean that I need some other strategy for the files. I could combine this approach
with the default in memory based store. But let just stick to the directories:
>
> I read the directory sorted by file name. The IdempotentRepository is
> called by the FtpConsumer with
>
> - start()
> - contains() for every directory and file
> - add() for files only
>
> and that's it. No stop(), no confirm(). When I have errors, sometimes remove() is called.
Since the repository is called only with String (the full path) I have not information if
I deal with directories or files. I know it from the structure, but I am not able to implement
a generic solution.
>
> Anyway the idea is:
>
> - Store the LastProcessedDate inside the repository
> - contains(): if the path contains an already processed date (<LastProcessedDate)
then I skip it (return true) otherwise return false.
> - add(): if add() jumps to the next directory I set the
> LastProcessedDate to the directory before
>
> The only problem is the last processed directory: even if it is finished I do not get
the chance to mark it as processed (set LastProcessedDate to its value).
>
> So finally my questions: do you think this approach makes sense and if yes: how would
you deal with the last processed directory?
> If no, how would you solve it?
>
> Thanks and kind regards, Christian



--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Mime
View raw message