datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Hayes <matthew.terence.ha...@gmail.com>
Subject Fwd: Hourglass Input paths
Date Wed, 05 Feb 2014 17:20:11 GMT
Moving discussion to datafu dev mailing list.

---------- Forwarded message ----------
From: Matthew Hayes <matthew.terence.hayes@gmail.com>
Date: Wed, Feb 5, 2014 at 9:18 AM
Subject: Re: Hourglass Input paths
To: "datafu@googlegroups.com" <datafu@googlegroups.com>


Hi Abhishek,

At the moment the input directory structure is fixed to the yyyy/mm/dd
format.  This is one area where I'd like to make the system more
customizable.  I'd like to create an interface that represents the concept
of a set of input paths that each are associated with a particular date
range, but without being tied to a particular format like yyyy/mm/dd.  Then
you would have the option of creating a class that implements this
interface and providing this class to Hourglass.  You could create a class
like WeeklyInput for example that assumes yyyy/mm/dd structure but dates
only on week boundaries.  Or you could create a class like MonthlyInput
that uses yyyy/mm.  Or one that uses yyyymmdd.  The same could be done for
the output path.  You could even have an input structure that has
overlapping date ranges, like "daily/yyyy/mm/dd" and "monthly/yyyy/mm".
Hourglass could determine the appropriate files to read based on the
desired date range to consume.  These are just some ideas.  I haven't had
time to work on this yet.  If you're interested in working on it we would
appreciate the help ;)

-Matt


On Wed, Feb 5, 2014 at 3:06 AM, Abhishek Gayakwad <a.gayakwad@gmail.com>wrote:

> Hi,
>
> Hourglass requires input directory structure in yyyy/mm/dd format, Is it
> possible to provide custom logic for reading from different directory
> structure e.g. yyyymmdd or some other date key. Is it possible to change
> the frequency form daily to weekly ?
>
> Thanks
> Abhishek
>
> --
> You received this message because you are subscribed to the Google Groups
> "DataFu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datafu+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message