Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Sender: sjlee0@gmail.com
In-Reply-To: <CAGi=jZC7DUcR=wKPLS7s-rGDZUOWKc=uTuTNv4jZp6dfQ8k=jw@mail.gmail.com>
References: <CAOPW3nNF4J_mdf=N79-7UzX=Z8PisoYgWdDAJ1AHWkWxr7vpbA@mail.gmail.com>
 <CAE=b_fZ+8UL-y_xcVfDb0T-NJWRfiUjZ+RHJmfoDY5Lr7K73ZA@mail.gmail.com>
 <CAGB5D2annn9hgpWiHMygBLy3LU1teyZ9P671qLDXj53Qrtrt8Q@mail.gmail.com>
 <CAOPW3nPiN6LtYfa7tpUzz8oY_OH1zMAeRTwN-ONxb_LHytZkDw@mail.gmail.com>
 <CAGB5D2aV33jX5QajkJhOgiQ5Qe=1Ht_m+D2xYOBfKqzVreaD6w@mail.gmail.com>
 <CADbBEnuaOJo1VT55q5zUMUKO_sN7TAmQ_F0XQH8J8u+R7v46UQ@mail.gmail.com> <CAGi=jZC7DUcR=wKPLS7s-rGDZUOWKc=uTuTNv4jZp6dfQ8k=jw@mail.gmail.com>
From: Sangjin Lee <sjlee@apache.org>
Date: Mon, 21 Aug 2017 17:16:30 -0700
Message-ID: <CADbBEnv6cCvnB574aN8Rq5WZRGSQEPLzdt+Kfs+RyjbtbPnc6w@mail.gmail.com>
Subject: Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
To: Rohith Sharma K S <rohithsharmaks@apache.org>
Cc: Andrew Wang <andrew.wang@cloudera.com>, Vrushali Channapattan <vrushali.c@gmail.com>,
	Vrushali Channapattan <vrushalic2016@gmail.com>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c1c054006827d05574c835b"

--94eb2c1c054006827d05574c835b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.

On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharmaks@apache.org> wrote:

> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well=
.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee <sjlee@apache.org> wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <andrew.wang@cloudera.com=
>
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushali.c@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at wh=
at
> > we
> > > > consider moderate scale but this did not include the security relat=
ed
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge a=
re
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We wi=
ll
> > > then
> > > > create a new branch with the new jira name and move these open jira=
s
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Shou=
ld
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs =
to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to fle=
x
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2016@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > > >>> resending
> > > >>> the same email with hopefully better looking content:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> I'd like to open a discussion for merging Timeline Service v2
> > > (YARN-5355)
> > > >>> to trunk in a few weeks.
> > > >>>
> > > >>> We have previously completed one merge onto trunk [1] and Timelin=
e
> > > >>> Service
> > > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>>
> > > >>> Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> Service v2 in a feature branch [2].  There are a few related issu=
es
> > > >>> pending
> > > >>> that are being actively worked upon and tested. As soon as they a=
re
> > > >>> resolved, we plan on starting a merge vote within the next two
> weeks.
> > > The
> > > >>> goal is to get this into hadoop3 beta.
> > > >>>
> > > >>> We have paid close attention to ensure that  once disabled Timeli=
ne
> > > >>> Service
> > > >>> v2 does not impact existing functionality when disabled (by
> default).
> > > >>>
> > > >>> At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> since 3.0.0-alpha1:
> > > >>> - Security (via Kerberos Authentication & delegation tokens)
> > > [YARN-3053]
> > > >>> - Timeline server usability improvements [timeline-server
> > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=3Dproject
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > > >>> 2C%20created%20ASC>
> > > >>> ]
> > > >>> - HBase specific improvements [atsv2-hbase
> > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=3Dproject
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> ]
> > > >>> - REST API additions and improvements [timeline-reader
> > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=3Dproject
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> > 20created%20ASC>
> > > >>> ]
> > > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > > >>>
> > > >>> We would love to get your thoughts on this before we open a real
> > voting
> > > >>> thread.
> > > >>>
> > > >>> Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>>
> > > >>> Thanks
> > > >>> Vrushali
> > > >>> [1] Merge to trunk: http://www.mail-archive.com/
> > yarn-dev@hadoop.apache
> > > .
> > > >>> org/msg23897.html
> > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg23897.html>
> > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > > >>> ache/hadoop/commits/YARN-5355
> > > >>>
> > > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > > >>> vrushali.c@gmail.com> wrote:
> > > >>>
> > > >>> > Hi All
> > > >>> >
> > > >>> > I=E2=80=99d like to open a discussion for merging Timeline Serv=
ice v.2
> > > >>> (YARN-5355)
> > > >>> > to trunk in a few weeks.
> > > >>> >
> > > >>> > We have previously completed one merge onto trunk [1] and
> Timeline
> > > >>> Service
> > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>> >
> > > >>> > Since then, we have been working on extending the capabilities =
of
> > > >>> Timeline
> > > >>> > Service v2 in a feature branch [2]. There are a few related
> issues
> > > >>> pending
> > > >>> > that are being actively worked upon & tested. As soon as they a=
re
> > > >>> resolved,
> > > >>> > we plan on starting a merge vote within the next 2 weeks. The
> goal
> > is
> > > >>> to
> > > >>> > get this in for Hadoop3 beta.
> > > >>> >
> > > >>> > We have paid close attention to ensure that once disabled
> Timeline
> > > >>> Service
> > > >>> > v.2 does not impact existing functionality when disabled (by
> > > default).
> > > >>> >
> > > >>> > At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> > since 3.0.0-alpha1:
> > > >>> > - Security (via Kerberos Authentication & delegation tokens) at
> the
> > > >>> writer
> > > >>> > [YARN-3053]
> > > >>> > - Timeline server usability improvements [timeline-server
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=3D
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> > ]
> > > >>> > - HBase specific improvements [atsv2-hbase
> > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=3D
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > > >>> ed%20DESC%2C%
> > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > > >>> > ]
> > > >>> > - REST API additions & improvements [timeline-reader
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=3D
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > > >>> ty%20DESC%2C%
> > > >>> > 20created%20ASC>
> > > >>> > ]
> > > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > > >>> >
> > > >>> > We would love to get your thoughts on these and more before we
> > open a
> > > >>> real
> > > >>> > voting thread.
> > > >>> >
> > > >>> > Special thanks to a team of folks who worked hard and contribut=
ed
> > > >>> towards
> > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > > >>> Varun
> > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli=
,
> > Joep
> > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>> >
> > > >>> > Thanks!
> > > >>> > Vrushali
> > > >>> >
> > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop=
.
> > > >>> > apache.org/msg23897.html
> > > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > > >>> > apache/hadoop/commits/YARN-5355
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

--94eb2c1c054006827d05574c835b--