Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of mkwhit@gmail.com designates
 209.85.223.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BLU0-SMTP4219F598C4EB2A6339348BCAD0B0@phx.gbl>
References: 
 <CAH29n6MxNcb3ztOE0cLjr2wAm2GEoAWim3ARgeyToF39oY24Rg@mail.gmail.com>
 <CAA5C_putLNXePOC6kw9nNQpMj3wSa_G6u-fK21mguwq4NvQg7Q@mail.gmail.com>
 <CAH29n6MySfhT9ZwW2e7bB6S+Qnf3pRU691gHUhHL0MtC_ytjyA@mail.gmail.com>
 <BLU0-SMTP135014842D8FE3881EE124CAD0B0@phx.gbl>
 <CAH29n6N2yDG0DpNbrXhU=Ngd4DM9h12PFaGeGgaNtC+D5Y62Ag@mail.gmail.com>
 <BLU0-SMTP4219F598C4EB2A6339348BCAD0B0@phx.gbl>
From: Micah Whitacre <mkwhit@gmail.com>
Date: Thu, 31 Oct 2013 07:25:36 -0500
Message-ID: 
 <CAKFfB9na3zz4O0zLqw8rAngwhxJQZgr++y_nS5jXpDhfY+uY9w@mail.gmail.com>
Subject: Re: Crunch release plan: 0.8.0 and HBase 0.96
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary=90e6ba614a7631ccf404ea088d4f

--90e6ba614a7631ccf404ea088d4f
Content-Type: text/plain; charset=ISO-8859-1

I'm in the same boat and will probably be on 0.94 for a few months.  So
will probably help with the backporting.


On Thu, Oct 31, 2013 at 2:19 AM, Chao Shi <stepinto@live.com> wrote:

> >> Will we release the hadoop1 jars as well?
> > I think that it's still worth doing; what about you?
>
> Agree, it would be nice to have it.
>
> 2013/10/31 Josh Wills <jwills@cloudera.com>
>
> > On Wed, Oct 30, 2013 at 11:17 PM, Chao Shi <stepinto@live.com> wrote:
> >
> > > Due to the client API changes and upgrade downtime, I can expect that
> > there
> > > are still a lot of users staying with 0.94. For our use case, we will
> > > continue to use 0.94 for at least several months. So I may still work
> on
> > > "0.8.1, 0.8.2..." if there are new features requests or urgent bugs to
> > fix.
> > >
> >
> > That would be fantastic.
> >
> >
> > >
> > > >> 4) Do the 0.9.0 release against with Hadoop 2 and HBase 0.96 as the
> > > default.
> > > Will we release the hadoop1 jars as well?
> > >
> >
> > I think that it's still worth doing; what about you?
> >
> > >
> > >
> > > 2013/10/31 Josh Wills <jwills@cloudera.com>
> > >
> > > > On Wed, Oct 30, 2013 at 2:23 PM, Gabriel Reid <
> gabriel.reid@gmail.com
> > > > >wrote:
> > > >
> > > > > Inlined below.
> > > > >
> > > > > On Wed, Oct 30, 2013 at 6:02 PM, Josh Wills <jwills@cloudera.com>
> > > wrote:
> > > > > >
> > > > > > That said, the
> > > > > > code changes that we'll need to make to get Crunch working
> against
> > > the
> > > > > 0.96
> > > > > > APIs are different enough from the 0.94 APIs that I feel like
> > > > maintaining
> > > > > > some sort of compatibility layer in our code will be pretty ugly.
> > > > >
> > > > > Yep, that's definitely something we don't want to do unless we've
> got
> > > > > a good reason
> > > > > to do it.
> > > >
> > > >
> > > > > >
> > > > > > I'm thinking along these lines:
> > > > > >
> > > > > > 1) Release 0.8.0 in the next couple of days against our current
> set
> > > of
> > > > > > dependencies (Hadoop and HBase.)
> > > > > > 2) Upgrade the Hadoop 2 dependency to Hadoop 2.2.0, which will
> also
> > > > > require
> > > > > > us to upgrade to protocol buffers 2.5.0 in the build-- I've
> already
> > > > done
> > > > > > this and verified that everything works.
> > > > > > 3) Switch the HBase code to the 0.96 APIs, without trying to
> > maintain
> > > > > > backwards compatibility with 0.94, and get everything working.
> > > > > > 4) Do the 0.9.0 release against with Hadoop 2 and HBase 0.96 as
> the
> > > > > default.
> > > > > >
> > > > > > I imagine that there will still be bugfixes against 0.8.0 (both
> > core
> > > > and
> > > > > > HBase) that will mean that we'll need to do 0.8.1, 0.8.2, etc.
> > > releases
> > > > > to
> > > > > > support, and I'm happy to keep those up at a regular cadence.
> > > > >
> > > > > This works for me, but I wish we had a better idea of what the
> > adoption
> > > > of
> > > > > HBase 0.96 will be. I'm guessing it'll be pretty high, as people
> who
> > > are
> > > > > just
> > > > > using the normal client APIs have a less troublesome migration path
> > > than
> > > > > those working with MapReduce. On the other hand, it would be a
> bummer
> > > to
> > > > > shut out all the 0.94.x users if there isn't major adoption of 0.96
> > > right
> > > > > away.
> > > > >
> > > > > Anyhow, like I said, I'm personally fine with just supporting 0.96
> > as I
> > > > > don't
> > > > > think it'll be a problem for me.
> > > > >
> > > >
> > > > Yeah, it's a hard balance to strike. I fully expect that we will have
> > > > 0.8.1, 0.8.2, etc. releases to bring some of the fixes we do in trunk
> > to
> > > > the HBase 0.94-based Crunch, which will still be the major version
> for
> > > > awhile. The HBase folks consider 0.96 the future and the best version
> > to
> > > > use w/Hadoop 2.2.0, so I'd like to pay whatever cost we have to pay
> in
> > > > terms of APIs and dependency changes all at once instead of
> piecemeal.
> > > >
> > > >
> > > > > - Gabriel
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Director of Data Science
> > > > Cloudera <http://www.cloudera.com>
> > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > >
> > >
> >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>

--90e6ba614a7631ccf404ea088d4f--