hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Fabbri <fab...@cloudera.com.INVALID>
Subject Re: Hadoop 3.2 Release Plan proposal
Date Tue, 02 Oct 2018 18:44:08 GMT
Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
top-level pom.xml?


On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org> wrote:

> Hi All
>
> As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> patches are pulled into branch-3.2.
>
> Thank You
> - Sunil
>
>
> On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org> wrote:
>
> > Hi All
> >
> > We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> > Thanks for the support.
> >
> > *Plan for RC*
> > 3.2 branch cut and reset trunk : *25th Tuesday*
> > RC0 for 3.2: *28th Friday*
> >
> > Thank You
> > Sunil
> >
> >
> > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org> wrote:
> >
> >> Hi All
> >>
> >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> >> helping in this. I am following up on these tickets, once its closed we
> >> will cut the 3.2 branch.
> >>
> >> Thanks
> >> Sunil Govindan
> >>
> >>
> >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Inline with the original 3.2 communication proposal dated 17th July
> >>> 2018, I would like to provide more updates.
> >>>
> >>> We are approaching previously proposed code freeze date (September 14,
> >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> existing
> >>> trunk to 3.3 if there are no issues.
> >>>
> >>> *Current Release Plan:*
> >>> Feature freeze date : all features to merge by September 7, 2018.
> >>> Code freeze date : blockers/critical only, no improvements and
> >>> blocker/critical bug-fixes September 14, 2018.
> >>> Release date: September 28, 2018
> >>>
> >>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> >>> backport to 3.2 post branch cut.
> >>>
> >>> Here's an updated 3.2.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>>
> >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>> workloads Initial cut.
> >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> Scheduler.
> >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> >>> and CLI.
> >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
> >>>
> >>> 2. Features close to finish:
> >>>
> >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> - (Steve) S3a phase V. Close to commit.
> >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>>
> >>> 3. Tentative/Cancelled features for 3.2:
> >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>> ATSv2. Patch in progress.
> >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> >>> be done before Aug 2018.
> >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> Challenging as more discussions are on-going.
> >>>
> >>> *Summary of 3.2.0 issues status:*
> >>> 19 Blocker and Critical issues [1] are open, I am following up with
> >>> owners to get status on each of them to get in by Code Freeze date.
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> >>> BY priority DESC
> >>>
> >>> Thanks,
> >>> Sunil
> >>>
> >>>
> >>>
> >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> Inline with earlier communication dated 17th July 2018, I would like
> to
> >>>> provide some updates.
> >>>>
> >>>> We are approaching previously proposed code freeze date (Aug 31).
> >>>>
> >>>> One of the critical feature Node Attributes feature merge
> >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> more
> >>>> time. With regard to this, suggesting to push the feature/code freeze
> for 2
> >>>> more weeks to accommodate these jiras too.
> >>>>
> >>>> Proposing Updated changes in plan inline with this:
> >>>> Feature freeze date : all features to merge by September 7, 2018.
> >>>> Code freeze date : blockers/critical only, no improvements and
> >>>>  blocker/critical bug-fixes September 14, 2018.
> >>>> Release date: September 28, 2018
> >>>>
> >>>> If any features in branch which are targeted to 3.2.0, please reply
to
> >>>> this email thread.
> >>>>
> >>>> *Here's an updated 3.2.0 feature status:*
> >>>>
> >>>> 1. Merged & Completed features:
> >>>>
> >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>> workloads Initial cut.
> >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>> Scheduler.
> >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>> API and CLI.
> >>>>
> >>>> 2. Features close to finish:
> >>>>
> >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> >>>> Ongoing.
> >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>> ATSv2. Patch in progress.
> >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>> Storage. In progress.
> >>>>
> >>>> 3. Tentative features:
> >>>>
> >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Aug 2018.
> >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>> Challenging as more discussions are on-going.
> >>>>
> >>>> *Summary of 3.2.0 issues status:*
> >>>>
> >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> >>>> owners to get status on each of them to get in by Code Freeze date.
> >>>>
> >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>> 3.2.0 ORDER BY priority DESC
> >>>>
> >>>> Thanks,
> >>>> Sunil
> >>>>
> >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> 17th
> >>>>> July 2018, I would like to provide some updates.
> >>>>>
> >>>>> We are approaching previously proposed feature freeze date (Aug
21,
> >>>>> about 7 days from today).
> >>>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> >>>>> this email thread.
> >>>>> Steve has mentioned about the s3 features which will come close
to
> >>>>> Code Freeze Date (Aug 31st).
> >>>>>
> >>>>> *Here's an updated 3.2.0 feature status:*
> >>>>>
> >>>>> 1. Merged & Completed features:
> >>>>>
> >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>>> workloads Initial cut.
> >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>>>
> >>>>> 2. Features close to finish:
> >>>>>
> >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> >>>>> patches are all in, only one last
> >>>>> patch is in review state.
> >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>>> Scheduler. Close to commit.
> >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>>> API and CLI. 2 patches are pending
> >>>>> which will be closed by Feature freeze date.
> >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>>> ATSv2. Patch in progress.
> >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> >>>>> works.
> >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>>> Storage. In progress.
> >>>>>
> >>>>> 3. Tentative features:
> >>>>>
> >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> >>>>> to be done before Aug 2018.
> >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>>> Challenging as more discussions are on-going.
> >>>>>
> >>>>> *Summary of 3.2.0 issues status:*
> >>>>>
> >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> owners
> >>>>> to get status on each of them to get in by Code Freeze date.
> >>>>>
> >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>>> 3.2.0 ORDER BY priority DESC
> >>>>>
> >>>>> Thanks,
> >>>>> Sunil
> >>>>>
> >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org>
wrote:
> >>>>>
> >>>>>> Thanks Subru for the thoughts.
> >>>>>> One of the main reason for a major release is to push out critical
> >>>>>> features with a faster cadence to the users. If we are pulling
more
> and
> >>>>>> more different types of features to a minor release, that branch
> will
> >>>>>> become more destabilized and it may be tough to say that 3.1.2
is
> stable
> >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features
> in
> >>>>>> subsequent minor release.
> >>>>>> For few companies, it makes sense to push out these new features
> >>>>>> faster to make a reach to the users. Adding to the point to
the
> backporting
> >>>>>> issues, I agree that its a pain and we can workaround that with
> some git
> >>>>>> scripts. If we can make such scripts available to committers,
> backport will
> >>>>>> be seem-less across branches and we can achieve the faster release
> cadence
> >>>>>> also.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> - Sunil
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Sunil for volunteering to lead the release effort.
I am
> >>>>>>> generally
> >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
as feel we
> >>>>>>> already
> >>>>>>> have too many branches to be maintained. I already see many
commits
> >>>>>>> are in
> >>>>>>> different branches with no apparent rationale, for e.g:
3.1 has
> >>>>>>> commits
> >>>>>>> which are absent in 3.0 etc.
> >>>>>>>
> >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> production
> >>>>>>> setting so the cost of adding features should be minimal.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> -Subru
> >>>>>>>
> >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>>>>>> >
> >>>>>>> > Yes, important changes and features are much needed,
hence we
> will
> >>>>>>> be
> >>>>>>> > keeping the door open for them as possible. Also considering
few
> >>>>>>> more
> >>>>>>> > offline requests from other folks, I think extending
the
> timeframe
> >>>>>>> by
> >>>>>>> > couple of weeks makes sense (including a second RC
buffer) and
> >>>>>>> this should
> >>>>>>> > ideally help us to ship this by September itself.
> >>>>>>> >
> >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
as well)
> >>>>>>> >
> >>>>>>> > - Feature freeze date : all features to merge by August
21, 2018.
> >>>>>>> >
> >>>>>>> > - Code freeze date : blockers/critical only, no improvements
and
> >>>>>>> non
> >>>>>>> > blocker/critical
> >>>>>>> >
> >>>>>>> > bug-fixes  August 31, 2018.
> >>>>>>> >
> >>>>>>> > - Release date: September 15, 2018
> >>>>>>> >
> >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
your
> >>>>>>> questions.
> >>>>>>> >
> >>>>>>> > Thanks
> >>>>>>> > Sunil
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>> >
> >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
release, +1 for
> >>>>>>> that.
> >>>>>>> > >
> >>>>>>> > > To concerns from Steve,
> >>>>>>> > >
> >>>>>>> > > It is a good idea to keep the door open to get
important
> changes
> >>>>>>> /
> >>>>>>> > > features in before cutoff. I would prefer to keep
the proposed
> >>>>>>> release
> >>>>>>> > date
> >>>>>>> > > to make sure things can happen earlier instead
of last minute
> >>>>>>> and we all
> >>>>>>> > > know that releases are always get delayed :).
I'm also fine if
> >>>>>>> we want
> >>>>>>> > get
> >>>>>>> > > another several weeks time.
> >>>>>>> > >
> >>>>>>> > > Regarding of 3.3 release, I would suggest doing
that before
> >>>>>>> thanksgiving.
> >>>>>>> > > Do you think is it good or too early / late?
> >>>>>>> > >
> >>>>>>> > > Eric,
> >>>>>>> > >
> >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
YARN-8135 can
> >>>>>>> get merged
> >>>>>>> > > in time, we probably not need the YARN-8220.
> >>>>>>> > >
> >>>>>>> > > Sunil,
> >>>>>>> > >
> >>>>>>> > > Could u update
> >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>>>>>> > Roadmap
> >>>>>>> > > with proposed plan as well? We can fill feature
list first
> >>>>>>> before getting
> >>>>>>> > > consensus of time.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Wangda
> >>>>>>> > >
> >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>>>>>> <fabbri@cloudera.com.invalid
> >>>>>>> > >
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran
<
> >>>>>>> stevel@hortonworks.com>
> >>>>>>> > >> wrote:
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> <mailto:
> >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>>>>>> > >> >
> >>>>>>> > >> > I would also would like to take this
opportunity to come up
> >>>>>>> with a
> >>>>>>> > >> detailed
> >>>>>>> > >> > plan.
> >>>>>>> > >> >
> >>>>>>> > >> > - Feature freeze date : all features
should be merged by
> >>>>>>> August 10,
> >>>>>>> > >> 2018.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > <snip>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > Please let me know if I missed any features
targeted to 3.2
> >>>>>>> per this
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > Well there these big todo lists for S3
& S3Guard.
> >>>>>>> > >> >
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > There's a bigger bit of work coming on
for Azure Datalake
> Gen
> >>>>>>> 2
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>>>>>> > >> >
> >>>>>>> > >> > I don't think this is quite ready yet,
I've been doing work
> >>>>>>> on it, but
> >>>>>>> > >> if
> >>>>>>> > >> > we have a 3 week deadline, I'm going
to expect some timely
> >>>>>>> reviews on
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>>>>>> > >> >
> >>>>>>> > >> > I've uprated that to a blocker feature;
will review the S3 &
> >>>>>>> S3Guard
> >>>>>>> > >> JIRAs
> >>>>>>> > >> > to see which of those are blocking. Then
there are some
> >>>>>>> pressing
> >>>>>>> > "guave,
> >>>>>>> > >> > java 9 prep"
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >>  I can help with this part if you like.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > timeline. I would like to volunteer myself
as release
> manager
> >>>>>>> of 3.2.0
> >>>>>>> > >> > release.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > well volunteered!
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Yes, thank you for stepping up.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > I think this raises a good q: what timetable
should we have
> >>>>>>> for the
> >>>>>>> > >> 3.2. &
> >>>>>>> > >> > 3.3 releases; if we do want a faster
cadence, then having
> the
> >>>>>>> outline
> >>>>>>> > >> time
> >>>>>>> > >> > from the 3.2 to the 3.3 release means
that there's less
> >>>>>>> concern about
> >>>>>>> > >> > things not making the 3.2 dealine
> >>>>>>> > >> >
> >>>>>>> > >> > -Steve
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Good idea to mitigate the short deadline.
> >>>>>>> > >>
> >>>>>>> > >> -AF
> >>>>>>> > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message