Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CANyB6pu4ELzi70VzMzna1VGt_OaKntWXM+pn+Q_aHUjwD6NiJg@mail.gmail.com>
References: <CAE=b_fbLT2J+Ezb4wqdN_UwBiG1Sd5kpqGaw+9Br__zou5yNTQ@mail.gmail.com>
 <CACYiTuhumHKVSb1HZqfGzcWsVjjPBHA0wSUPBONrc4BH+bKeDw@mail.gmail.com>
 <CAP4BK9QiZqCZ_8kXOpX6Mbn+Az5W-M+MPk-q0J_33946TjBjWA@mail.gmail.com>
 <CADbBEns4+oXrmpuZFv3aFdVUV9UdW7To6fzrjY96OFLPwjfcwQ@mail.gmail.com>
 <CAOScs9aWaTeTv2cnibf8nHzyx-io_Yd2KNZk9s2QxRRgsrFLLg@mail.gmail.com> <CANyB6pu4ELzi70VzMzna1VGt_OaKntWXM+pn+Q_aHUjwD6NiJg@mail.gmail.com>
From: Vrushali C <vrushalic2016@gmail.com>
Date: Fri, 25 Aug 2017 15:05:55 -0700
Message-ID: <CAE=b_fZ23nY+OmHJSeSNf2FAS9J5tJ9q8UjF_5cg4=tf9ww1vw@mail.gmail.com>
Subject: Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
To: varunsaxena@apache.org
Cc: Subramaniam V K <subru.vk@gmail.com>, Sangjin Lee <sjlee@apache.org>,
	Haibo Chen <haibochen@cloudera.com>, Sunil G <sunilg@apache.org>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c1a3ed037a57305579b2683"
archived-at: Fri, 25 Aug 2017 22:06:06 -0000

--94eb2c1a3ed037a57305579b2683
Content-Type: text/plain; charset="UTF-8"

Hi Subru,

Thanks for your vote and your response!

Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.

thanks
Vrushali


On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:

> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <subru.vk@gmail.com>
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sjlee@apache.org> wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <haibochen@cloudera.com>
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <sunilg@apache.org> wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2016@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > > In a nutshell, Timeline Service v.2 delivers significant
>> scalability
>> > > and
>> > > > > usability improvements based on a new architecture. What we would
>> > like
>> > > to
>> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has
>> a
>> > > > > complete end-to-end read/write flow with security and read level
>> > > > > authorization via whitelists. You should be able to start setting
>> it
>> > up
>> > > > and
>> > > > > testing it.
>> > > > >
>> > > > > At a high level, the following are the key features that have been
>> > > > > implemented since alpha1:
>> > > > > - Security via Kerberos Authentication and delegation tokens
>> > > > > - Read side simple authorization via whitelist
>> > > > > - Client configurable entity sort ordering
>> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
>> > metrics
>> > > > by
>> > > > > timerange, pagination, sub-app entities
>> > > > > - Support for storing sub-application entities (entities that
>> exist
>> > > > outside
>> > > > > the scope of an application)
>> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
>> > > > prefixes,
>> > > > > configurable hbase cluster
>> > > > > - Flow level aggregations done as dynamic (table level)
>> coprocessors
>> > > > > - Uses latest stable HBase release 1.2.6
>> > > > >
>> > > > > There are a total of 82 subtasks that were completed as part of
>> this
>> > > > > effort.
>> > > > >
>> > > > > We paid close attention to ensure that once disabled Timeline
>> Service
>> > > v.2
>> > > > > does not impact existing functionality when disabled (by default).
>> > > > >
>> > > > > Special thanks to a team of folks who worked hard and contributed
>> > > towards
>> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
>> > > Varun
>> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> Joep
>> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > > > >
>> > > > > Regards,
>> > > > > Vrushali
>> > > > >
>> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> > > 383.html
>> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
>> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
>> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

--94eb2c1a3ed037a57305579b2683--