incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bayer <andrew.ba...@gmail.com>
Subject Re: Deciding on the Hadoop version for Bigtop 0.2.0
Date Fri, 30 Sep 2011 20:29:15 GMT
I vote for 1, for the simple reason that 2 and 3, as Bruno says, are simply
not viable options. We can't release with 0.22 if the full stack won't work
with it, and we can't release with 0.20.205.0 if we can't even build it.

I really think it's important to get 0.2.0 out in the next few weeks - we've
made a ton of progress from 0.1.0, especially in the testing realm, and I
think that alone merits a release. 0.1.0 wasn't actually usable - 0.2.0, as
it currently stands, *could* be usable. That's a big deal. That said, I
don't think there's any problem at all with continuing what we've done
already, with pre-released 0.22 work on a branch, letting us provide a venue
for testing interoperability between the stack components before release.
We'll do the same with 0.23, as I see it. If 0.22 and the stack stabilize
in, say, late October/early November, I'd be all for pushing an 0.2.1
release incorporating the 0.22-compatible stack, with 0.3.0 targeting the
0.23 compatible stack.

The biggest interest in Bigtop so far seems to be in terms of stack testing,
so I think it makes sense to put an emphasis on that, distinct from the
releases. We should release only a fully functional stack, providing
packages and a general baseline for what's the stable Hadoop stack is at
this time. But we should also be doing what Roman's been doing, working on
testing pre-released versions of the stack for interop, general QA, being
ahead of the curve on build/packaging changes, etc. The releases serve
Hadoop stack users, while the pre-release testing work serves the Hadoop
stack development community, and feeds into the releases.

A.

On Fri, Sep 30, 2011 at 12:31 PM, Bruno Mahé <bmahe@apache.org> wrote:

> On 09/30/2011 12:06 PM, Roman Shaposhnik wrote:
> > Hi!
> >
> > By now it is obvious that the decision on what version of Hadoop to use
> > for Bigtop 0.2.0 is not going to be an easy one ;-)
> >
> > So far we've got the following choices:
> >    1. Do nothing and stick with 0.20.2. This is not all that bad, I
> suppose,
> >        we've got tons of updates to the Bigtop to justify a release. But
> it
> >        feels a little bit anticlimactic in a sense that Bigtop 0.3.0 is
> very
> >        likely to be on top of 0.23 and I really would like to have a
> chance
> >        of having a Bigtop release on top of the last MR1.
> >    2. Target upcoming Hadoop 0.22. This is going to break all of the
> >        downstream except for HBase. Here's a list of JIRAs filed:
> >            https://issues.apache.org/jira/browse/SQOOP-354
> >            https://issues.apache.org/jira/browse/PIG-2277
> >            https://issues.apache.org/jira/browse/OOZIE-565
> >            https://issues.apache.org/jira/browse/HIVE-2468
> >            https://issues.apache.org/jira/browse/MAHOUT-822
> >         As you can see, most of the downstream is OKins with
> >         implementing the changes in time for .23, but not necessarily
> >         in time for .22 and Bigtop 0.2.0.
> >    3. Target 0.20.205.0. The downside there is that unless the following
> >         build issues are resolved, we can't even compile it on the
> platforms
> >         Bigtop cares about:
> >              https://issues.apache.org/jira/browse/HADOOP-6436
> >              https://issues.apache.org/jira/browse/MAPREDUCE-2127
> >              https://issues.apache.org/jira/browse/HDFS-2327
> >
> > For #2 and #3 we can, potentially, mitigate the timing issue by patching
> > the downstream locally at the Bigtop level. Do we want to entertain such
> > an idea?
> >
> > Anyway, please chime in with your thoughts. It would be very nice to
> > have Bigtop 0.2.0 fully tested and out sometime around first weeks of
> > Nov. As such we have to make a decision on Hadoop rather soon.
> >
> > Thanks,
> > Roman.
>
> Nothing prevents us to do multiple releases :)
> No matter which version of hadoop we decide to jump to, we should do a
> last release of BigTop with hadoop 0.20.2. As you said, we already have
> tons of updates to justify a release.
>
> But given the no patch policy, even for build or security issues, we
> can't jump to any release until we can get something that at least
> compile. So I would definitely entertain the idea of patching projects
> in BigTop for build and security issues only. Otherwise we get stuck in
> a chicken and egg problem where upstream projects would like to see some
> testing done before releasing, but we can't test because there is no
> release. As well as the effort to convince each project to accept
> patches and do releases compatible with versions used in BigTop.
> IMHO, most GNU/Linux distributions have a nice patch policy where each
> patch must also be attached to a ticket in upstream projects. So
> upstream projects can benefit from it, but won't block downstream
> progress and everyone can focus on their goal and get things done.
>
> But for now, neither 2. nor 3. appear to be an option.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message