asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Jacobs <sjaco...@ucr.edu>
Subject Re: UPDATE: FileSystemFeedAdapter [Design]
Date Fri, 14 Aug 2015 15:34:53 GMT
I would like to as well.
Steven

On Friday, August 14, 2015, Murtadha Hubail <hubailmor@gmail.com> wrote:

> I would be happy to review it.
>
> Regards,
> Murtadha
>
> > On Aug 14, 2015, at 10:34 AM, abdullah alamoudi <amoudi@apache.org
> <javascript:;>> wrote:
> >
> > Hi,
> > I just want to give a little update about what I have been up to
> recently.
> > I have completed the design and implementation of the
> > FileSystemFeedAdapter. The adapter basically gets a list of directories
> as
> > a parameter and start ingesting records in files that exists within the
> > specified directories.
> >
> > I am using the Java WatchService API to monitor the selected directories.
> > With this API, the system receives three types of notifications:
> > 1. File Create notification. This notification is sent whenever a new
> file
> > is created or a file has been renamed.
> > 2. File Delete notification. This notification is sent whenever an
> existing
> > file is deleted or a file has been renamed.
> > 3. File Modification notification. This notification is sent whenever
> > changes to an existing file are saved.
> >
> > We would like to avoid reading files which are being written and so we
> > added another parameter that the user is expected to supply,
> > "matching-expression". This expression is matched against File Create
> > notifications and will start immediately ingesting data from the file.
> The
> > user is expected to write their data in a file with a name that doesn't
> > match the expression (i.e., something that ends with .tmp, .staging, etc)
> > and rename it to something that matches the expression when the file is
> > ready to be read. I think many other systems do the same.
> >
> > The feed stops working if all the watched directories have been removed
> or
> > when the user disconnect the feed.
> >
> > The only remaining piece of this is how we can address feed continuation
> in
> > case of system failure as right now, I am not storing progress on disk at
> > all. This however will be added in another iteration.
> >
> > A change request will be submitted for this today and I am thinking about
> > selecting @Murtadha, @Ian for review. Please, confirm that you have time
> to
> > do it :-) and thanks in advance :-). I would've picked @Raman but he
> seems
> > very busy these days.
> >
> > If any of this doesn't make sense or if there is a better way to do any
> of
> > this, please let me know.
> >
> > Cheers,
> > Abdullah.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message