hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin P. McCabe" <cmcc...@apache.org>
Subject Re: Looking to a Hadoop 3 release
Date Mon, 22 Feb 2016 23:19:21 GMT
I think starting a 3.0 alpha soon would be a great idea.  As some
other people commented, this would come with no compatibility
guarantees, so that we can iron out any issues.

Colin

On Mon, Feb 22, 2016 at 1:26 PM, Zhe Zhang <zhezhang@cloudera.com> wrote:
> Thanks Andrew for driving the effort!
>
> +1 (non-binding) on starting the 3.0 release process now with 3.0 as an
> alpha.
>
> I wanted to echo Andrew's point that backporting EC to branch-2 is a lot of
> work. Considering that no concrete backporting plan has been proposed, it
> seems quite uncertain whether / when it can be released in 2.9. I think we
> should rather concentrate our EC dev efforts to harden key features under
> the follow-on umbrella HDFS-8031 and make it solid for a 3.0 release.
>
> Sincerely,
> Zhe
>
> On Mon, Feb 22, 2016 at 9:25 AM Colin P. McCabe <cmccabe@apache.org> wrote:
>
>> +1 for a release of 3.0.  There are a lot of significant,
>> compatibility-breaking, but necessary changes in this release... we've
>> touched on some of them in this thread.
>>
>> +1 for a parallel release of 2.8 as well.  I think we are pretty close
>> to this, barring a dozen or so blockers.
>>
>> best,
>> Colin
>>
>> On Mon, Feb 22, 2016 at 2:56 AM, Steve Loughran <stevel@hortonworks.com>
>> wrote:
>> >
>> >> On 20 Feb 2016, at 15:34, Junping Du <jdu@hortonworks.com> wrote:
>> >>
>> >> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds
>> reasonable to have two alpha releases to go in parallel. Is EC feature the
>> main motivation of releasing hadoop 3 here? If so, I don't understand why
>> this feature cannot land on 2.8.x or 2.9.x as an alpha feature.
>> >
>> >
>> >
>> >> If we release 3.0 in a month like plan proposed below, it means we will
>> have 4 active releases going in parallel - two alpha releases (2.8 and 3.0)
>> and two stable releases (2.6.x and 2.7.x). It brings a lot of challenges in
>> issues tracking and patch committing, not even mention the tremendous
>> effort of release verification and voting.
>> >> I would like to propose to wait 2.8 release become stable (may be 2nd
>> release in 2.8 branch cause first release is alpha due to discussion in
>> another email thread), then we can move to 3.0 as the only alpha release.
>> In the meantime, we can bring more significant features (like ATS v2, etc.)
>> to trunk and consolidate stable releases in 2.6.x and 2.7.x. I believe that
>> make life easier. :)
>> >> Thoughts?
>> >>
>> >
>> > 2.8.0 is relatively close to shipping. I say relatively as I'm doing
>> some work with ATS 1.5 downstream and I'd like to make sure all that works.
>> There's also a large collection of S3 and swift patches needing attention
>> from any reviewers with time and credentials.
>> >
>> > 3.x is going to take multiple iterations to stabilise, and with more
>> changes, more significant a rollout. I'd also like to do a complete update
>> of all the dependencies before a final release, so we can have less
>> pressure to upgrade for a while, and get Sean's classloader patch in so
>> it's slightly less visible.
>> >
>> > That means 3.0 is going to be an alpha release, not final.
>> >
>> > one thing that could be shared is any build.xml automation of the
>> release process, to at least take away most of the manual steps in the
>> process, to have something more repeatable.
>> >
>> > -steve
>> >
>> >
>> >> Thanks,
>> >>
>> >> Junping
>> >> ________________________________________
>> >> From: Yongjun Zhang <yzhang@cloudera.com>
>> >> Sent: Friday, February 19, 2016 8:05 PM
>> >> To: hdfs-dev@hadoop.apache.org
>> >> Cc: common-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
>> yarn-dev@hadoop.apache.org
>> >> Subject: Re: Looking to a Hadoop 3 release
>> >>
>> >> Thanks Andrew for initiating the effort!
>> >>
>> >> +1 on pushing 3.x with extended alpha cycle, and continuing the more
>> stable
>> >> 2.x releases.
>> >>
>> >> --Yongjun
>> >>
>> >> On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang <andrew.wang@cloudera.com>
>> >> wrote:
>> >>
>> >>> Hi Kai,
>> >>>
>> >>> Sure, I'm open to it. It's a new major release, so we're allowed to
>> make
>> >>> these kinds of big changes. The idea behind the extended alpha cycle
is
>> >>> that downstreams can give us feedback. This way if we do anything too
>> >>> radical, we can address it in the next alpha and have downstreams
>> re-test.
>> >>>
>> >>> Best,
>> >>> Andrew
>> >>>
>> >>> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <kai.zheng@intel.com>
>> wrote:
>> >>>
>> >>>> Thanks Andrew for driving this. Wonder if it's a good chance for
>> >>>> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in.
Note
>> >>> it's
>> >>>> not an incompatible change, but feel better to be done in the major
>> >>> release.
>> >>>>
>> >>>> Regards,
>> >>>> Kai
>> >>>>
>> >>>> -----Original Message-----
>> >>>> From: Andrew Wang [mailto:andrew.wang@cloudera.com]
>> >>>> Sent: Friday, February 19, 2016 7:04 AM
>> >>>> To: hdfs-dev@hadoop.apache.org; Kihwal Lee <kihwal@yahoo-inc.com>
>> >>>> Cc: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org;
>> >>>> yarn-dev@hadoop.apache.org
>> >>>> Subject: Re: Looking to a Hadoop 3 release
>> >>>>
>> >>>> Hi Kihwal,
>> >>>>
>> >>>> I think there's still value in continuing the 2.x releases. 3.x
comes
>> >>> with
>> >>>> the incompatible bump to a JDK8 runtime, and also the fact that
3.x
>> won't
>> >>>> be beta or GA for some number of months. In the meanwhile, it'd
be
>> good
>> >>> to
>> >>>> keep putting out regular, stable 2.x releases.
>> >>>>
>> >>>> Best,
>> >>>> Andrew
>> >>>>
>> >>>>
>> >>>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee
>> <kihwal@yahoo-inc.com.invalid
>> >>>>
>> >>>> wrote:
>> >>>>
>> >>>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main
>> >>>>> motivations, are we getting rid of branch-2.8?
>> >>>>>
>> >>>>> Kihwal
>> >>>>>
>> >>>>>      From: Andrew Wang <andrew.wang@cloudera.com>
>> >>>>> To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
>> >>>>> Cc: "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>;
"
>> >>>>> mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>;
>> >>>>> hdfs-dev <hdfs-dev@hadoop.apache.org>
>> >>>>> Sent: Thursday, February 18, 2016 4:35 PM
>> >>>>> Subject: Re: Looking to a Hadoop 3 release
>> >>>>>
>> >>>>> Hi all,
>> >>>>>
>> >>>>> Reviving this thread. I've seen renewed interest in a trunk
release
>> >>>>> since HDFS erasure coding has not yet made it to branch-2. Along
with
>> >>>>> JDK8, the shell script rewrite, and many other improvements,
I think
>> >>>>> it's time to revisit Hadoop 3.0 release plans.
>> >>>>>
>> >>>>> My overall plan is still the same as in my original email: a
series
>> of
>> >>>>> regular alpha releases leading up to beta and GA. Alpha releases
make
>> >>>>> it easier for downstreams to integrate with our code, and making
them
>> >>>>> regular means features can be included when they are ready.
>> >>>>>
>> >>>>> I know there are some incompatible changes waiting in the wings
(i.e.
>> >>>>> HDFS-6984 making FileStatus a PB rather than Writable, some
of
>> >>>>> HADOOP-9991 bumping dependency versions) that would be good
to get
>> in.
>> >>>>> If you have changes like this, please set the target version
to 3.0.0
>> >>>>> and mark them "Incompatible". We can use this JIRA query to
track:
>> >>>>>
>> >>>>>
>> >>>>>
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
>> >>>>>
>> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
>> >>>>>
>> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
>> >>>>> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
>> >>>>>
>> >>>>> There's some release-related stuff that needs to be sorted out
>> >>>>> (namely, the new CHANGES.txt and release note generation from
Yetus),
>> >>>>> but I'd tentatively like to roll the first alpha a month out,
so
>> third
>> >>>>> week of March.
>> >>>>>
>> >>>>> Best,
>> >>>>> Andrew
>> >>>>>
>> >>>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata <rstata@altiscale.com>
>> >>>> wrote:
>> >>>>>
>> >>>>>> Avoiding the use of JDK8 language features (and, presumably,
APIs)
>> >>>>>> means you've abandoned #1, i.e., you haven't (really) bumped
the JDK
>> >>>>>> source version to JDK8.
>> >>>>>>
>> >>>>>> Also, note that releasing from trunk is a way of achieving
#3, it's
>> >>>>>> not a way of abandoning it.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
>> >>>>>> <andrew.wang@cloudera.com>
>> >>>>>> wrote:
>> >>>>>>> Hi Raymie,
>> >>>>>>>
>> >>>>>>> Konst proposed just releasing off of trunk rather than
cutting a
>> >>>>>> branch-2,
>> >>>>>>> and there was general agreement there. So, consider
#3 abandoned.
>> >>>>>>> 1&2
>> >>>>> can
>> >>>>>>> be achieved at the same time, we just need to avoid
using JDK8
>> >>>>>>> language features in trunk so things can be backported.
>> >>>>>>>
>> >>>>>>> Best,
>> >>>>>>> Andrew
>> >>>>>>>
>> >>>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
>> >>>>>>> <rstata@altiscale.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> In this (and the related threads), I see the following
three
>> >>>>>> requirements:
>> >>>>>>>>
>> >>>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop
JDK7 support).
>> >>>>>>>>
>> >>>>>>>> 2. "We'll still be releasing 2.x releases for a
while, with
>> >>>>>>>> similar feature sets as 3.x."
>> >>>>>>>>
>> >>>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize
>> >>>>>>>> backporting headaches. Pulling trunk > branch-2
> branch-2.x is
>> >>>> already tedious.
>> >>>>>>>> Adding a branch-3, branch-3.x would be obnoxious."
>> >>>>>>>>
>> >>>>>>>> These three cannot be achieved at the same time.
 Which do we
>> >>>> abandon?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia
>> >>>>>>>> <sanjayosrc@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth
<sseth@apache.org>
>> >>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> 2) Simplification of configs - potentially
separating client
>> >>>>>>>>>> side
>> >>>>>>>> configs
>> >>>>>>>>>> and those used by daemons. This is another
source of perpetual
>> >>>>>> confusion
>> >>>>>>>>>> for users.
>> >>>>>>>>> + 1 on this.
>> >>>>>>>>>
>> >>>>>>>>> sanjay
>> >>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>

Mime
View raw message