crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <m...@mafr.de>
Subject Re: New module to share user functions
Date Tue, 25 Sep 2012 18:29:28 GMT
Hi Rahul,

I think it would be really great to have an ecosystem of
micro-libraries around Crunch for all kinds of cool stuff that is
relevant for smaller audiences, just like your Bloom filters.

But since I expect most of this stuff to be so extremely special, it
would in my opinion make more sense to put this into small, focused
and independent projects that can be released separately from each
other and don't need to go through Crunch's review process. It would
make dependency management easier for users, too, in case a library
needs additional dependencies.

We could maintain a registry of these projects on Crunch's homepage
so people can find them easily (I expect most of them would end up
at GitHub because it's perfect for this kind of thing). If a project
turns out to be interesting for a larger audience, we can still add it
to Crunch core.

Regards,
  Matthias

On Tuesday, 2012-09-25, Rahul wrote:
> There can be interesting use-cases like BloomFilters which do not
> have a place in the current set of Crunch modules. These functions
> are kind of utility functions that can be used in Crunch. We need to
> create a place where users can share such functions. In the earlier
> discussion for BloomFilters we thought of some thing that is well
> along the lines of PiggyBank. I had a look at the module but in
> Pig's structure the module is branched under contrib module as there
> are other modules like peeny for monitering and zebra for storage.
> 
> I have created a module name *crunch-bytes* , for issue
> https://issues.apache.org/jira/browse/CRUNCH-75, which is direct
> sub-module in crunch-parent. I named it so because I felt it will
> providing a space to have all those interesting data computations
> that we can not have in core.
> 
> Please share your thoughts for the same.
> 
> regards,
> rahul
> 

Mime
View raw message