hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Templeton <dan...@cloudera.com>
Subject Re: Hadoop 3.1.0 release discussion
Date Wed, 31 Jan 2018 08:35:36 GMT
I added my comments on that JIRA.  Looks like YARN-7292 is marked as a 
blocker for 3.1, and I would tend to agree with that.  Let's see what we 
can to do get profiles nailed down so that 3.1 can go forward.

Daniel

On 1/18/18 10:25 AM, Wangda Tan wrote:
> Thanks Daniel,
>
> We need to make a decision about this: 
> https://issues.apache.org/jira/browse/YARN-7292, I believe this is the 
> JIRA you mentioned correct? Please let me know if there's anything 
> else. And let's move the discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so 
> we can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <daniel@cloudera.com 
> <mailto:daniel@cloudera.com>> wrote:
>
>     What's the status on resource profiles?  I believe there are still
>     a couple of open JIRAs to rethink some of the design choices.
>
>     Daniel
>
>
>     On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>         Hi All,
>
>         Since we're fast approaching previously proposed feature
>         freeze date (Jan
>         30, about 13 days from today). If you've any features which
>         live in a
>         branch and targeted to 3.1.0, please reply this email thread.
>         Ideally, we
>         should finish branch merging before feature freeze date.
>
>         Here's an updated 3.1.0 feature status:
>
>         1. Merged & Completed features:
>         * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>         * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
>         and works
>         end-to-end.
>         * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>         services.
>         * (Steve Loughran): HADOOP-13786: S3Guard committer for
>         zero-rename commits.
>         * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
>         of Leaf
>         Queues While Doing Queue Mapping.
>         * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>         2. Features close to finish:
>         * (Zhankun) YARN-5983: FPGA support. Majority implementations
>         completed and
>         merged to trunk. Except for UI/documentation.
>         * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>         done, some
>         discussions going on about implementation.
>         * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>         SchedulingRequest and
>         anti-affinity support. Close to finish, on track to be merged
>         before Jan 30.
>
>         3. Tentative features:
>         * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>         containers. Only one pending patch. Plan to finish before Jan 7th.
>         * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>         challenging to be
>         done before Jan 2018.
>         * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
>         Looks
>         challenging to be done before Jan 2018.
>         * (Varun V) YARN-5673: container-executor write. Given
>         security refactoring
>         of c-e (YARN-6623) is already landed, IMHO other stuff may be
>         moved to 3.2.
>
>         Thanks,
>         Wangda
>
>
>
>
>         On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan
>         <wheeleast@gmail.com <mailto:wheeleast@gmail.com>> wrote:
>
>             Hi all,
>
>             Congratulations on the 3.0.0-GA release!
>
>             As we discussed in the previous email thread [1], I'd like
>             to restart
>             3.1.0 release plans.
>
>             a) Quick summary:
>             a.1 Release status
>             We started 3.1 release discussion on Sep 6, 2017 [1]. As
>             of today,
>             there’re 232 patches loaded on 3.1.0 alone [2], besides 6
>             open blockers and
>             22 open critical issues.
>
>             a.2 Release date update
>             Considering delays of 3.0-GA release by month-and-a-half,
>             I propose to
>             move the dates as follows
>               - feature freeze date from Dec 15, 2017, to Jan 30, 2018
>             - last date for
>             any branches to get merged too;
>               - code freeze (blockers & critical only) date to Feb 08,
>             2018;
>               - release voting start by Feb 18, 2018, leaving time for
>             at least two RCx
>               - release date from Jan 15, 2018, to Feb 28, 2018;
>
>             Unlike before, I added an additional milestone for
>             release-vote-start so
>             that we can account for voting time-period also.
>
>             This overall is still 5 1/2 months of release-timeline
>             unlike the faster
>             cadence we hoped for, but this, in my opinion, is the
>             best-updated timeline
>             given the delays of the final release of 3.0-GA.
>
>             b) Individual feature status:
>             I spoke to several feature owners and checked the status
>             of un-finished
>             features, following are status of features planned to 3.1.0:
>
>             b.1 Merged & Completed features:
>             * (Sunil) YARN-5881: Support absolute value in
>             CapacityScheduler.
>             * (Wangda) YARN-6223: GPU support on YARN. Features in
>             trunk and works
>             end-to-end.
>             * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
>             native services.
>             * (Steve Loughran): HADOOP-13786: S3Guard committer for
>             zero-rename
>             commits.
>             * (Suma): YARN-7117: Capacity Scheduler: Support Auto
>             Creation of Leaf
>             Queues While Doing Queue Mapping.
>
>             b.2 Features close to finish:
>             * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
>             voting now.
>             * (Zhankun) YARN-5983: FPGA support. Majority
>             implementations completed
>             and merged to trunk. Except for UI/documentation.
>             * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>             done, some
>             discussions going on about implementation.
>
>             b.3 Tentative features:
>             * (Arun Suresh). YARN-5972: Support pausing/freezing
>             opportunistic
>             containers. Only one pending patch. Plan to finish before
>             Jan 7th.
>             * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>             challenging to
>             be done before Jan 2018.
>             * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>             SchedulingRequest and
>             anti-affinity support. Tentative will figure out by Jan 1st.
>             * (Anu): HDFS-7240: Ozone. Given the discussion on
>             HDFS-7240. Looks
>             challenging to be done before Jan 2018.
>             * (Varun V) YARN-5673: container-executor write. Given
>             security
>             refactoring of c-e (YARN-6623) is already landed, IMHO
>             other stuff may be
>             moved to 3.2.
>
>             b.4 Additional release drivers
>             * More exhaustive upgrade testing from 2.x to 3.x.
>
>             c) Regarding branch cut:
>
>             We will keep pointing trunk to 3.1 and cut branch-3.1
>             until: A. some
>             feature planned to 3.2 has to be landed on trunk or B.
>             After feature freeze
>             date, whichever comes first.
>
>             I've also talked offline with Vinod to get help on
>             release-management
>             given this is my first release. He agreed to help do this
>             release jointly.
>
>             Thoughts?
>
>             Thanks,
>             Wangda Tan
>
>             [1]
>             https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>             <https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0>
>             9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
>             <http://3Cyarn-dev.hadoop.apache.org>%3E
>             [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>             fixVersion in (3.1.0)
>             AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>     <mailto:common-dev-unsubscribe@hadoop.apache.org>
>     For additional commands, e-mail: common-dev-help@hadoop.apache.org
>     <mailto:common-dev-help@hadoop.apache.org>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message