incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Focus of our next release?
Date Fri, 14 Sep 2012 18:33:06 GMT
On Fri, Sep 14, 2012 at 11:30 AM, Joseph Adler <joseph.adler@gmail.com>wrote:

> It could be interesting to do the streaming on top of Apache Kafka because
> both systems work well with Avro serialization.
>

Good point-- I like the idea of supporting Apache projects wherever
possible-- hcat and solr being great examples of this. I'd love some
thoughts on Crunch-for-Kafka from you and the rest of the Kafka community.


> On Fri, Sep 14, 2012 at 11:17 AM, Josh Wills <jwills@cloudera.com> wrote:
>
> > I like the idea of having themes for releases. In my head, the theme of
> > this release could be either
> >
> > a) Hacking the new MSCRPlanner code, esp. to add the ability to fuse
> > different MSCR jobs into a single instance that it enables, or
> > b) data access/integration points-- things like solr, hcatalog, hbase,
> > cassandra, jdbc, etc. as input and output sources for Crunch pipelines,
> or
> > c) API refactoring-- the crunch-api/crunch-impl/crunch-lib split, or
> > d) working on a PStream API that would let people apply DoFns to streams
> > and would build on top of things like WalMart's mupd8 or Storm or
> whatever.
> >
> > Of course, this is in addition to whatever fixes and new lib functions we
> > want to add over time. I don't want anything heavyweight, but those are
> > some of the larger-scale things that we'll need to tackle as a community,
> > and I would think of completing each of those big things as corresponding
> > to a release.
> >
> > Just my two cents.
> >
> > J
> >
> > On Fri, Sep 14, 2012 at 10:23 AM, Matthias Friedrich <matt@mafr.de>
> wrote:
> >
> > > Hi,
> > >
> > > should we discuss the focus of our next release? Maybe make a list
> > > of things we want to achieve? Or would this be too much process?
> > >
> > > Regards,
> > >   Matthias
> > >
> >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message