hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subru Krishnan <su...@apache.org>
Subject [DISCUSS] 2.9+ stabilization branch
Date Mon, 26 Feb 2018 22:37:18 GMT

We (i.e. Microsoft) have started stabilization of 2.9 for our production
deployment. During planning, we realized that we need to backport 3.x
features to support GPUs (and more resource types like network IO) natively
as part of the upgrade. We'd like to share that work with the community.

Instead of stabilizing the base release and cherry-picking fixes back to
Apache, we want to work publicly and push fixes directly into
trunk/.../branch-2 for a stable 2.10.0 release. Our goal is to create a
bridge release for our production clusters to the 3.x series and to address
scalability problems in large clusters (N*10k nodes). As we find issues, we
will file JIRAs and track resolution of significant regressions/faults in
wiki. Moreover, LinkedIn also has committed plans for a production
deployment of the same branch. We welcome broad participation, particularly
since we'll be stabilizing relatively new features.

The exact list of features we would like to backport in YARN are:

   - Support for Resource types [1][2]
   - Native support for GPUs[3]
   - Absolute Resource configuration in CapacityScheduler [4]

With regards to HDFS, we are currently looking at mainly fixes to Router
based Federation and Windows specific fixes which should anyways flow



[1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27786.html
[2] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28281.html
[3] https://issues.apache.org/jira/browse/YARN-6223
[4] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28772.html

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message