hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@apache.org>
Subject Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Date Sat, 19 Aug 2017 10:34:23 GMT
Hi Sangjin

Thanks for bringing this point.
We did similar exercise with current YARN-5355 branch today. Tests are
validated against default configuration and timeline server v.1.5 as well.

All YARN major features such as RM HA/Restart/work-preserving-restart, NM
restart, scheduling, sample MR/distributes shell jobs run are validated.
Everything looks fine.

This testing will also be done once YARN-5355 branch code freezed
completely as well in couple of days.

Thanks & Regards
Rohith Sharma K S

On 18 August 2017 at 22:56, Sangjin Lee <sjlee@apache.org> wrote:

> Kudos to Vrushali and the team for getting ready for this large and
> important feature! I know a huge team effort went into this. I look forward
> to seeing this merged.
>
> I'd like to ask one piece of due diligence. Could you please inspect
> rigorously to ensure that when disabled Timeline Service v.2 does not
> impact other features in any way? We did a similar exercise when we had the
> first drop, and it would be good to repeat that... Thanks!
>
> Sangjin
>
> On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <andrew.wang@cloudera.com>
> wrote:
>
> > Great, thanks Vrushali! Sounds good to me.
> >
> > I have a few procedural release notes comments I'll put on YARN-5355, to
> > make sure we advertise this to our users appropriately.
> >
> > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > vrushali.c@gmail.com> wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your response!
> > >
> > > There have been no changes to existing APIs since alpha1.
> > >
> > > We at Twitter have tested the feature to demonstrate it works at what
> we
> > > consider moderate scale but this did not include the security related
> > > testing. The security testing is in progress at present by Timeline
> > Service
> > > V2 team in the community and we think we will have more details on this
> > > very soon.
> > >
> > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > > think of as "merge-blockers". The issues being targeted for merge are
> in
> > > [link1] below. There are about 59 jiras of which 56 are completed.
> > >
> > > We plan to make a new umbrella jira after the merge to trunk. We will
> > then
> > > create a new branch with the new jira name and move these open jiras
> > under
> > > YARN-5355 as subtasks of that new umbrella jira.
> > >
> > > thanks
> > > Vrushali
> > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> > >
> > >
> > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > wrote:
> > >
> > >> Hi Vrushali,
> > >>
> > >> Glad to hear this major dev milestone is nearing completion!
> > >>
> > >> Repeating my request on other merge [DISCUSS] threads, could you
> comment
> > >> on testing and API stability of this merge? Our timeline for beta1 is
> > about
> > >> a month out, so there's not much time to fix things beforehand.
> > >>
> > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > most
> > >> of these be moved out to a new umbrella? I'm wondering what needs to
> be
> > >> completed before sending the merge vote.
> > >>
> > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> the
> > >> beta1 release date for this feature than others. Hopefully that won't
> be
> > >> necessary though :)
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > >>> resending
> > >>> the same email with hopefully better looking content:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> I'd like to open a discussion for merging Timeline Service v2
> > (YARN-5355)
> > >>> to trunk in a few weeks.
> > >>>
> > >>> We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> Service v2 in a feature branch [2].  There are a few related issues
> > >>> pending
> > >>> that are being actively worked upon and tested. As soon as they are
> > >>> resolved, we plan on starting a merge vote within the next two weeks.
> > The
> > >>> goal is to get this into hadoop3 beta.
> > >>>
> > >>> We have paid close attention to ensure that  once disabled Timeline
> > >>> Service
> > >>> v2 does not impact existing functionality when disabled (by default).
> > >>>
> > >>> At a high level, following are the key features that have been
> > >>> implemented
> > >>> since 3.0.0-alpha1:
> > >>> - Security (via Kerberos Authentication & delegation tokens)
> > [YARN-3053]
> > >>> - Timeline server usability improvements [timeline-server
> > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > >>> 2C%20created%20ASC>
> > >>> ]
> > >>> - HBase specific improvements [atsv2-hbase
> > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> ]
> > >>> - REST API additions and improvements [timeline-reader
> > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > >>> ]
> > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > >>>
> > >>> We would love to get your thoughts on this before we open a real
> voting
> > >>> thread.
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Thanks
> > >>> Vrushali
> > >>> [1] Merge to trunk: http://www.mail-archive.com/
> yarn-dev@hadoop.apache
> > .
> > >>> org/msg23897.html
> > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg23897.html>
> > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > >>> ache/hadoop/commits/YARN-5355
> > >>>
> > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > >>> vrushali.c@gmail.com> wrote:
> > >>>
> > >>> > Hi All
> > >>> >
> > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > >>> (YARN-5355)
> > >>> > to trunk in a few weeks.
> > >>> >
> > >>> > We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >
> > >>> > Since then, we have been working on extending the capabilities
of
> > >>> Timeline
> > >>> > Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> > that are being actively worked upon & tested. As soon as they
are
> > >>> resolved,
> > >>> > we plan on starting a merge vote within the next 2 weeks. The
goal
> is
> > >>> to
> > >>> > get this in for Hadoop3 beta.
> > >>> >
> > >>> > We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> > v.2 does not impact existing functionality when disabled (by
> > default).
> > >>> >
> > >>> > At a high level, following are the key features that have been
> > >>> implemented
> > >>> > since 3.0.0-alpha1:
> > >>> > - Security (via Kerberos Authentication & delegation tokens)
at the
> > >>> writer
> > >>> > [YARN-3053]
> > >>> > - Timeline server usability improvements [timeline-server
> > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - HBase specific improvements [atsv2-hbase
> > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > >>> ed%20DESC%2C%
> > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - REST API additions & improvements [timeline-reader
> > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > >>> ty%20DESC%2C%
> > >>> > 20created%20ASC>
> > >>> > ]
> > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > >>> >
> > >>> > We would love to get your thoughts on these and more before we
> open a
> > >>> real
> > >>> > voting thread.
> > >>> >
> > >>> > Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> > this effort with patches, reviews and guidance: Rohith Sharma
K S,
> > >>> Varun
> > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >
> > >>> > Thanks!
> > >>> > Vrushali
> > >>> >
> > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > >>> > apache.org/msg23897.html
> > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > >>> > apache/hadoop/commits/YARN-5355
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message