hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject Re: Hadoop 3.1.0 release discussion
Date Tue, 30 Jan 2018 14:23:32 GMT
Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <uma.gangumalla@intel.com>
wrote:

> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wheeleast@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wheeleast@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <uma.gangumalla@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vinodkv@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wheeleast@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wheeleast@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message