asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Maxon <ima...@uci.edu>
Subject Re: UPDATE: FileSystemFeedAdapter [Design]
Date Fri, 14 Aug 2015 15:53:38 GMT
Looks like you'll have three, then :)

On Fri, Aug 14, 2015 at 8:34 AM, Steven Jacobs <sjaco002@ucr.edu> wrote:

> I would like to as well.
> Steven
>
> On Friday, August 14, 2015, Murtadha Hubail <hubailmor@gmail.com> wrote:
>
> > I would be happy to review it.
> >
> > Regards,
> > Murtadha
> >
> > > On Aug 14, 2015, at 10:34 AM, abdullah alamoudi <amoudi@apache.org
> > <javascript:;>> wrote:
> > >
> > > Hi,
> > > I just want to give a little update about what I have been up to
> > recently.
> > > I have completed the design and implementation of the
> > > FileSystemFeedAdapter. The adapter basically gets a list of directories
> > as
> > > a parameter and start ingesting records in files that exists within the
> > > specified directories.
> > >
> > > I am using the Java WatchService API to monitor the selected
> directories.
> > > With this API, the system receives three types of notifications:
> > > 1. File Create notification. This notification is sent whenever a new
> > file
> > > is created or a file has been renamed.
> > > 2. File Delete notification. This notification is sent whenever an
> > existing
> > > file is deleted or a file has been renamed.
> > > 3. File Modification notification. This notification is sent whenever
> > > changes to an existing file are saved.
> > >
> > > We would like to avoid reading files which are being written and so we
> > > added another parameter that the user is expected to supply,
> > > "matching-expression". This expression is matched against File Create
> > > notifications and will start immediately ingesting data from the file.
> > The
> > > user is expected to write their data in a file with a name that doesn't
> > > match the expression (i.e., something that ends with .tmp, .staging,
> etc)
> > > and rename it to something that matches the expression when the file is
> > > ready to be read. I think many other systems do the same.
> > >
> > > The feed stops working if all the watched directories have been removed
> > or
> > > when the user disconnect the feed.
> > >
> > > The only remaining piece of this is how we can address feed
> continuation
> > in
> > > case of system failure as right now, I am not storing progress on disk
> at
> > > all. This however will be added in another iteration.
> > >
> > > A change request will be submitted for this today and I am thinking
> about
> > > selecting @Murtadha, @Ian for review. Please, confirm that you have
> time
> > to
> > > do it :-) and thanks in advance :-). I would've picked @Raman but he
> > seems
> > > very busy these days.
> > >
> > > If any of this doesn't make sense or if there is a better way to do any
> > of
> > > this, please let me know.
> > >
> > > Cheers,
> > > Abdullah.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message