hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Vavilapalli <vino...@hortonworks.com>
Subject Re: [DISCUSS] Looking to a 2.8.0 release
Date Wed, 11 Nov 2015 20:13:55 GMT
Agreed on not mixing this with major release discussions.

Okay, I just finished my review of 2.8 content.

A quick summary follows.

Current state of originally planned items

 - Nearly Done / Done and so need to close down quickly
    — Support *both* JDK7 and JDK8 runtimes HADOOP-11090
    — Supporting non-exclusive node-labels: YARN-3214: Done, can push as an alpha / beta
feature
    — Support priorities across applications within the same queue YARN-1963: Can push as
an alpha / beta feature

 - Definitely have to move out into 2.9 and beyond
    — Early work for disk and network isolation in YARN: YARN-2139, YARN-2140: Early noise,
some critical pieces designed, done but not a lot of movement of late
    — Classpath isolation for downstream clients HADOOP-11656: Lots of chatter a while ago,
not much movement of late
    — Support for Erasure Codes in HDFS HDFS-7285<https://issues.apache.org/jira/browse/HDFS-7285>:
Moved out to 2.9 in the interest of stability / bake-in

Non-planned features that went into 2.8.0

    — The overall list of new features: https://issues.apache.org/jira/issues/?filter=12333994
    — HDFS-6200 Create a separate jar for hdfs-client: Compatible improvement - no dimension
of alpha/betaness here.
    — HDFS-8155    Support OAuth2 in WebHDFS: Alpha / Early feature?
    — Stability improvements ready to use:
        — HDFS-8008    Support client-side back off when the datanodes are congested
        — HDFS-8009    Signal congestion on the DataNode
    — YARN-3611 Support Docker Containers In LinuxContainerExecutor: Well most of it anyways.
Can push as an alpha feature.
    — YARN-1197 Support changing resources of an allocated container: Can push as an alpha/beta
feature

Items in progress to think about in 2 weeks

    — YARN Timeline Service v1.5 - YARN-4233: A short term bridge before YARN-2928 comes
around. I think this should go in given the tremendous activity recently.
    — YARN Timeline Service Next generation: YARN-2928: Lots of momentum, but clearly a
work in progress. Two options here
        — If it is safe to ship it into 2.8 in a disable manner, we can get the early code
into trunk and all the way int o2.8.
        — If it is not safe, it organically rolls over into 2.9
    — Compatibility tools to catch backwards, forwards compatibility issues at patch submission,
release times. Some of it is captured at YARN-3292. This also involves resurrecting jdiff
(HADOOP-11776/YARN-3426/MAPREDUCE-6310) and/or investing in new tools.

This is my plan of action for now in terms of the release itself

 - Cut a branch about two weeks from now
 - Do an RC mid next month (leaving ~4weeks since branch-cut)
 - As with 2.7.x series, the first release will still be called as early / alpha release in
the interest of
    — gaining downstream adoption
    — wider testing,
    — yet reserving our right to fix any inadvertent incompatibilities introduced.

If we can get answers on “Items to think about now” during this and next week, we will
overall be in good shape.

Thoughts?

Thanks
+Vinod
PS:As you may have noted above, this time around, I want to do something that we’ve always
wanted to do, but never explicitly did. I’m calling out readiness of each feature as they
stand today so we can inform our users better of what they can start relying on in production
clusters.


On Oct 5, 2015, at 11:53 AM, Colin P. McCabe <cmccabe@apache.org<mailto:cmccabe@apache.org>>
wrote:

I think it makes sense to have a 2.8 release since there are a
tremendous number of JIRAs in 2.8 that are not in 2.7.  Doing a 3.x
release seems like something we should consider separately since it
would not have the same compatibility guarantees as a 2.8 release.
There's a pretty big delta between trunk and 2.8 as well.

cheers,
Colin

On Sat, Sep 26, 2015 at 1:36 PM, Chris Douglas <cdouglas@apache.org<mailto:cdouglas@apache.org>>
wrote:
With two active sustaining branches (2.6, 2.7), what would you think
of releasing trunk as 3.x instead of pushing 2.8? There are many new
features (EC, Y1197, etc.), and trunk could be the source of several
alpha/beta releases before we fork the 3.x line. -C

On Sat, Sep 26, 2015 at 12:49 PM, Vinod Vavilapalli
<vinodkv@hortonworks.com<mailto:vinodkv@hortonworks.com>> wrote:
As you may have noted, 2.8.0 got completely derailed what with 2.7.x and the unusually long
2.6.1 release.

With 2.6.1 out of the way, and two parallel threads in progress for 2.6.2 and 2.7.2, it’s
time for us to look back at where we are with Hadoop 2.8.

I’ll do a quick survey of where the individual features are and the amount of content already
present in 2.8 and kick-start 2.8.0 process again.

+Vinod


On Apr 21, 2015, at 2:39 PM, vinodkv@apache.org<mailto:vinodkv@apache.org> wrote:

With 2.7.0 out of the way, and with more maintenance releases to stabilize it, I propose we
start thinking about 2.8.0.

Here's my first cut of the proposal, will update the Roadmap wiki.
- Support *both* JDK7 and JDK8 runtimes: HADOOP-11090
- Compatibility tools to catch backwards, forwards compatibility issues at patch submission,
release times. Some of it is captured at YARN-3292. This also involves resurrecting jdiff
(HADOOP-11776/YARN-3426/MAPREDUCE-6310) and/or investing in new tools.
- HADOOP-11656 Classpath isolation for downstream clients
- Support for Erasure Codes in HDFS HDFS-7285
- Early work for disk and network isolation in YARN: YARN-2139, YARN-2140
- YARN Timeline Service Next generation: YARN-2928. At least branch-merge + early peek.
- Supporting non-exclusive node-labels: YARN-3214

I'm experimenting with more agile 2.7.x releases and would like to continue the same by volunteering
as the RM for 2.8.x too.

Given the long time we took with 2.7.0, the timeline I am looking at is 8-12 weeks. We can
pick as many features as they finish along and make a more predictable releases instead of
holding up releases for ever.

Thoughts?

Thanks
+Vinod



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message