hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Lowe <jl...@yahoo-inc.com.INVALID>
Subject Re: Looking to a Hadoop 3 release
Date Thu, 05 Mar 2015 21:44:57 GMT
I'm OK with a 3.0.0 release as long as we are minimizing the pain of maintaining yet another
release line and conscious of the incompatibilities going into that release line.
For the former, I would really rather not see a branch-3 cut so soon.  It's yet another line
onto which to cherry-pick, and I don't see why we need to add this overhead at such an early
phase.  We should only create branch-3 when there's an incompatible change that the community
wants and it should _not_ go into the next major release (i.e.: it's for Hadoop 4.0).  We
can develop 3.0 alphas and betas on trunk and release from trunk in the interim.  IMHO we
need to stop treating trunk as a place to exile patches.

For the latter, I think as a community we need to evaluate the benefits of breaking compatibility
against the costs of migrating.  Each time we break compatibility we create a hurdle for
people to jump when they move to the new release, and we should make those hurdles worth their
time.  For example, wire-compatibility has been mentioned as part of this.  Any feature
that breaks wire compatibility better be absolutely amazing, as it creates a huge hurdle for
people to jump.
To summarize:+1 for a community-discussed roadmap of what we're breaking in Hadoop 3 and why
it's worth it for users
-1 for creating branch-3 now, we can release from trunk until the next incompatibility for
Hadoop 4 arrives
+1 for baking classpath isolation as opt-in on 2.x and eventually default on in 3.0
Jason
      From: Andrew Wang <andrew.wang@cloudera.com>
 To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org> 
Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org"
<mapreduce-dev@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>

 Sent: Wednesday, March 4, 2015 12:15 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Let's not dismiss this quite so handily.

Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
could make classpath isolation opt-in via configuration, what we really
want longer term is to have it on by default (or just always on). Stack in
particular points out the practical difficulties in using an opt-in method
in 2.x from a downstream project perspective. It's not pretty.

The plan that both Sean and Jason propose (which I support) is to have an
opt-in solution in 2.x, bake it there, then turn it on by default
(incompatible) in a new major release. I think this lines up well with my
proposal of some alphas and betas leading up to a GA 3.x. I'm also willing
to help with 2.x release management if that would help with testing this
feature.

Even setting aside classpath isolation, a new major release is still
justified by JDK8. Somehow this is being ignored in the discussion. Allen,
historically the voice of the user in our community, just highlighted it as
a major compatibility issue, and myself and Tucu have also expressed our
very strong concerns about bumping this in a minor release. 2.7's bump is a
unique exception, but this is not something to be cited as precedent or
policy.

Where does this resistance to a new major release stem from? As I've
described from the beginning, this will look basically like a 2.x release,
except for the inclusion of classpath isolation by default and target
version JDK8. I've expressed my desire to maintain API and wire
compatibility, and we can audit the set of incompatible changes in trunk to
ensure this. My proposal for doing alpha and beta releases leading up to GA
also gives downstreams a nice amount of time for testing and validation.

Regards,
Andrew



On Tue, Mar 3, 2015 at 2:32 PM, Arun Murthy <acm@hortonworks.com> wrote:

> Awesome, looks like we can just do this in a compatible manner - nothing
> else on the list seems like it warrants a (premature) major release.
>
> Thanks Vinod.
>
> Arun
>
> ________________________________________
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
> Sent: Tuesday, March 03, 2015 2:30 PM
> To: common-dev@hadoop.apache.org
> Cc: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> I started pitching in more on that JIRA.
>
> To add, I think we can and should strive for doing this in a compatible
> manner, whatever the approach. Marking and calling it incompatible before
> we see proposal/patch seems premature to me. Commented the same on JIRA:
> https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=14345875&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14345875
> .
>
> Thanks
> +Vinod
>
> On Mar 2, 2015, at 8:08 PM, Andrew Wang <andrew.wang@cloudera.com<mailto:
> andrew.wang@cloudera.com>> wrote:
>
> Regarding classpath isolation, based on what I hear from our customers,
> it's still a big problem (even after the MR classloader work). The latest
> Jackson version bump was quite painful for our downstream projects, and the
> HDFS client still leaks a lot of dependencies. Would welcome more
> discussion of this on HADOOP-11656, Steve, Colin, and Haohui have already
> chimed in.
>
>


  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message