hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hung <jyhung2...@gmail.com>
Subject Re: [DISCUSS] Looking to a 2.9.0 release
Date Tue, 05 Sep 2017 23:26:48 GMT
Hi Subru,

Thanks for starting the discussion. We are targeting merging YARN-5734
(API-based scheduler configuration) to branch-2 before the release of
2.9.0, since the feature is close to complete. Regarding the requirements
for merge,

1. API compatibility - this feature adds new APIs, does not modify any
existing ones.
2. Turning feature off - using the feature is configurable and is turned
off by default.
3. Stability/testing - this is an RM-only change, so we plan on deploying
this feature to a test RM and verifying configuration changes for capacity
scheduler. (Right now fair scheduler is not supported.)
4. Deployment - we want to get this feature in to 2.9.0 since we want to
use this feature and 2.9 version in our next upgrade.
5. Timeline - we have one main blocker which we are planning to resolve by
end of week. The rest of the month will be testing then a merge vote on the
last week of Sept.

Please let me know if you have any concerns. Thanks!


Jonathan Hung

On Wed, Jul 26, 2017 at 11:23 AM, J. Rottinghuis <jrottinghuis@gmail.com>
wrote:

> Thanks Vrushali for being entirely open as to the current status of ATSv2.
> I appreciate that we want to ensure things are tested at scale, and as you
> said we are working on that right now on our clusters.
> We have tested the feature to demonstrate it works at what we consider
> moderate scale.
>
> I think the criteria for including this feature in the 2.9 release should
> be if it can be safely turned off and not cause impact to anybody not using
> the new feature. The confidence for this is high for timeline service v2.
>
> Therefore, I think timeline service v2 should definitely be part of 2.9.
> That is the big draw for us to work on stabilizing a 2.9 release rather
> than just going to 2.8 and back-porting things ourselves.
>
> Thanks,
>
> Joep
>
> On Tue, Jul 25, 2017 at 11:39 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Thanks Subru for initiating this discussion.
> >
> > Wanted to share some thoughts in the context of Timeline Service v2. The
> > current status of this module is that we are ramping up for a second
> merge
> > to trunk. We still have a few merge blocker jiras outstanding, which we
> > think we will finish soon.
> >
> > While we have done some testing, we are yet to test at scale. Given all
> > this, we were thinking of initially targeting a beta release vehicle
> rather
> > than a stable release.
> >
> > As such, timeline service v2 has branch-2 branch called as
> > YARN-5355-branch-2 in case anyone wants to try it out. Timeline service
> v2
> > can be turned off and should not affect the cluster.
> >
> > thanks
> > Vrushali
> >
> >
> >
> >
> >
> > On Mon, Jul 24, 2017 at 1:26 PM, Subru Krishnan <subru@apache.org>
> wrote:
> >
> > > Folks,
> > >
> > > With the release for 2.8, we would like to look ahead to 2.9 release as
> > > there are many features/improvements in branch-2 (about 1062 commits),
> > that
> > > are in need of a release vechile.
> > >
> > > Here's our first cut of the proposal from the YARN side:
> > >
> > >    1. Scheduler improvements (decoupling allocation from node
> heartbeat,
> > >    allocation ID, concurrency fixes, LightResource etc).
> > >    2. Timeline Service v2
> > >    3. Opportunistic containers
> > >    4. Federation
> > >
> > > We would like to hear a formal list from HDFS & Hadoop (& MapReduce
if
> > any)
> > > and will update the Roadmap wiki accordingly.
> > >
> > > Considering our familiarity with the above mentioned YARN features, we
> > > would like to volunteer as the co-RMs for 2.9.0.
> > >
> > > We want to keep the timeline at 8-12 weeks to keep the release
> pragmatic.
> > >
> > > Feedback?
> > >
> > > -Subru/Arun
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message