asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Murtadha Hubail <hubail...@gmail.com>
Subject Re: UPDATE: FileSystemFeedAdapter [Design]
Date Fri, 14 Aug 2015 11:31:05 GMT
I would be happy to review it.

Regards,
Murtadha

> On Aug 14, 2015, at 10:34 AM, abdullah alamoudi <amoudi@apache.org> wrote:
> 
> Hi,
> I just want to give a little update about what I have been up to recently.
> I have completed the design and implementation of the
> FileSystemFeedAdapter. The adapter basically gets a list of directories as
> a parameter and start ingesting records in files that exists within the
> specified directories.
> 
> I am using the Java WatchService API to monitor the selected directories.
> With this API, the system receives three types of notifications:
> 1. File Create notification. This notification is sent whenever a new file
> is created or a file has been renamed.
> 2. File Delete notification. This notification is sent whenever an existing
> file is deleted or a file has been renamed.
> 3. File Modification notification. This notification is sent whenever
> changes to an existing file are saved.
> 
> We would like to avoid reading files which are being written and so we
> added another parameter that the user is expected to supply,
> "matching-expression". This expression is matched against File Create
> notifications and will start immediately ingesting data from the file. The
> user is expected to write their data in a file with a name that doesn't
> match the expression (i.e., something that ends with .tmp, .staging, etc)
> and rename it to something that matches the expression when the file is
> ready to be read. I think many other systems do the same.
> 
> The feed stops working if all the watched directories have been removed or
> when the user disconnect the feed.
> 
> The only remaining piece of this is how we can address feed continuation in
> case of system failure as right now, I am not storing progress on disk at
> all. This however will be added in another iteration.
> 
> A change request will be submitted for this today and I am thinking about
> selecting @Murtadha, @Ian for review. Please, confirm that you have time to
> do it :-) and thanks in advance :-). I would've picked @Raman but he seems
> very busy these days.
> 
> If any of this doesn't make sense or if there is a better way to do any of
> this, please let me know.
> 
> Cheers,
> Abdullah.


Mime
View raw message