crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <>
Subject Re: Focus of our next release?
Date Sat, 15 Sep 2012 09:55:43 GMT

On Friday, 2012-09-14, Josh Wills wrote:
> I like the idea of having themes for releases. In my head, the theme of
> this release could be either
> a) Hacking the new MSCRPlanner code, esp. to add the ability to fuse
> different MSCR jobs into a single instance that it enables, or
> b) data access/integration points-- things like solr, hcatalog, hbase,
> cassandra, jdbc, etc. as input and output sources for Crunch pipelines, or
> c) API refactoring-- the crunch-api/crunch-impl/crunch-lib split, or

I would see c) as part of a larger mission for improving documentation
and usability. An immediate benefit would be that we don't have to
provide javadoc for each and every class, only for those packages that
are client-facing. Higher perceived quality with less work for us.

I wouldn't make it a separate release though, perhaps we can do this
in a series of smaller steps, starting with the crunch-api split.
Refactorings like this usually turn into long frustrating monster
tasks that prevent other progress. I'd really like to avoid that.

But, before spending any more time on this, I think we should all
agree that this is what we really want. Somehow I got the impression
that you're not fully convinced that this refactoring is necessary or
even a good idea. To me it would feel like people trying to rearrange
the furniture in my living room. Let's discuss this here before we
produce any more patches.

> d) working on a PStream API that would let people apply DoFns to streams
> and would build on top of things like WalMart's mupd8 or Storm or whatever.
> Of course, this is in addition to whatever fixes and new lib functions we
> want to add over time. I don't want anything heavyweight, but those are
> some of the larger-scale things that we'll need to tackle as a community,
> and I would think of completing each of those big things as corresponding
> to a release.

Sounds good to me.


View raw message