datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Hayes <>
Subject Fwd: Hourglass Input paths
Date Wed, 05 Feb 2014 17:20:11 GMT
Moving discussion to datafu dev mailing list.

---------- Forwarded message ----------
From: Matthew Hayes <>
Date: Wed, Feb 5, 2014 at 9:18 AM
Subject: Re: Hourglass Input paths
To: "" <>

Hi Abhishek,

At the moment the input directory structure is fixed to the yyyy/mm/dd
format.  This is one area where I'd like to make the system more
customizable.  I'd like to create an interface that represents the concept
of a set of input paths that each are associated with a particular date
range, but without being tied to a particular format like yyyy/mm/dd.  Then
you would have the option of creating a class that implements this
interface and providing this class to Hourglass.  You could create a class
like WeeklyInput for example that assumes yyyy/mm/dd structure but dates
only on week boundaries.  Or you could create a class like MonthlyInput
that uses yyyy/mm.  Or one that uses yyyymmdd.  The same could be done for
the output path.  You could even have an input structure that has
overlapping date ranges, like "daily/yyyy/mm/dd" and "monthly/yyyy/mm".
Hourglass could determine the appropriate files to read based on the
desired date range to consume.  These are just some ideas.  I haven't had
time to work on this yet.  If you're interested in working on it we would
appreciate the help ;)


On Wed, Feb 5, 2014 at 3:06 AM, Abhishek Gayakwad <>wrote:

> Hi,
> Hourglass requires input directory structure in yyyy/mm/dd format, Is it
> possible to provide custom logic for reading from different directory
> structure e.g. yyyymmdd or some other date key. Is it possible to change
> the frequency form daily to weekly ?
> Thanks
> Abhishek
> --
> You received this message because you are subscribed to the Google Groups
> "DataFu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to
> For more options, visit

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message