hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Saxena <vsaxena.va...@gmail.com>
Subject Re: [YARN-2928] first drop on trunk
Date Tue, 24 Nov 2015 05:25:18 GMT
I think this JIRA should be complete by the time we merge it into trunk.

It would be nice to have the CLI work with ATSv2 if application is not
found in RM.

Regards,
Varun Saxena.

On Tue, 24 Nov 2015 at 10:48, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:

> Hi Sangjin,
>
>     Seems like the list is fine, can we have YARN-4343 too in the list ?
>
> + Naga
> ________________________________________
> From: Varun Saxena [vsaxena.varun@gmail.com]
> Sent: Saturday, November 14, 2015 01:46
> To: Sangjin Lee
> Cc: Joep Rottinghuis; Karthik Kambatla; yarn-dev@hadoop.apache.org; Li
> Lu; Junping Du; zjshen@apache.org; Vinod Kumar Vavilapalli; Naganarasimha
> Garla; Robert Kanter; Vrushali Channapattan
> Subject: Re: [YARN-2928] first drop on trunk
>
> The list
>
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%20yarn-2928-1st-milestone
> looks good to me.
>
> Regards,
> Varun Saxena.
>
>
> On Sat, Nov 14, 2015 at 12:09 AM, Sangjin Lee <sjlee@apache.org> wrote:
>
> > I took a stab at labeling JIRAs that I think we need for the first
> > milestone drop, and came up with 17 JIRAs:
> >
> https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
> >
> > Please take a look at them, and comment on them.
> >
> > Thanks,
> > Sangjin
> >
> > On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <joep@twitter.com>
> wrote:
> >
> >> Agreed with the theme and lists for the merge back of the first
> >> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
> >> suggestion with the two lists Sangjin.
> >>
> >> As we discussed in the weekly sync with the ATS v2 team, right after
> >> we've merged back, the next thing to decide is what the story will be to
> >> run both ATS v1 and ATS v2 in parallel so that we can start discussing
> when
> >> we can do a merge to branch-2.
> >> We'll want people (including ourselves) to be able to run ATS v2 on a
> >> trial basis w/o impacting existing use-cases on a branch-2 based
> release.
> >> The merge-back to branch-2 will be a separate topic of discussion
> though.
> >>
> >> Thanks,
> >>
> >> Joep
> >>
> >>
> >> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <kasha@cloudera.com>
> >> wrote:
> >>
> >>> +1
> >>>
> >>> In favor of merging a version working end-to-end at alpha quality to
> >>> trunk.
> >>>
> >>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sjlee@apache.org> wrote:
> >>>
> >>>> Just a reminder that I'd like input from folks that are working on
> this
> >>>> feature (as well as anyone who's interested or course!).
> >>>>
> >>>> I propose creating a label to start labeling things we'd like to see
> >>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
> >>>>
> >>>> Thanks,
> >>>> Sangjin
> >>>>
> >>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <
> vsaxena.varun@gmail.com
> >>>> > wrote:
> >>>>
> >>>>> Hi Sangjin,
> >>>>>
> >>>>> Thanks for initiating the discussion.
> >>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863
are
> >>>>> somewhat related.
> >>>>>
> >>>>> Regards,
> >>>>> Varun Saxena.
> >>>>>
> >>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sjlee@apache.org>
> wrote:
> >>>>>
> >>>>>> In the past several months, the timeline service v.2 team has
made a
> >>>>>> tremendous progress. We have a working storage implementation
based
> on
> >>>>>> HBase, timeline collectors, timeline readers with some of the
more
> >>>>>> important queries and filters, integration with RM and NM,
> distributed
> >>>>>> shell and mapreduce, and some basic UI to boot. We're getting
real
> close to
> >>>>>> a complete end-to-end flow (no pun intended). Kudos to the team
> (cc'ed
> >>>>>> here)!
> >>>>>>
> >>>>>> I think it is time to discuss defining a merge to trunk of an
> >>>>>> alpha-quality release as our first milestone so that a wider
> audience has a
> >>>>>> chance to try it out. This doesn't replace the timeline service
> (ATS) v.1
> >>>>>> yet, but it would be a great chance to get feedback.
> >>>>>>
> >>>>>> I think the theme is essentially a basic but complete end-to-end
> flow
> >>>>>> that includes the write path and the read path and some UI.
These
> are the
> >>>>>> key major things we may want to complete before we consider
merging
> the
> >>>>>> first milestone:
> >>>>>> - application aggregation (YARN-3816
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
> >>>>>> - flow run compaction work (YARN-4062
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
> >>>>>> - finalize the metrics storage (YARN-4053
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
> >>>>>> - improve queries and filters (YARN-3863
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
> >>>>>> - UI POC based on the new YARN UI framework (YARN-4097
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
> >>>>>>
> >>>>>> In addition to these, we would like to close a few more JIRAs
that
> >>>>>> we're currently working on. Also, in terms of the app integration,
> we can
> >>>>>> debate whether we stick to the distributed shell for now or
spend
> some more
> >>>>>> effort to round out the mapreduce support.
> >>>>>>
> >>>>>> I also think the following major things are probably out of
scope
> for
> >>>>>> this first drop:
> >>>>>> - time-based (offline) user/queue aggregation based on Phoenix
(
> >>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
> >>>>>> - fault-tolerant storage (YARN-4061
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
> >>>>>> - timeline collector as a separate daemon (YARN-3033
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
> >>>>>> - timeline collector containerization
> >>>>>> - compatibility with v.1 (YARN-3196
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
> >>>>>> - support for off-cluster timeline clients (YARN-3981
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
> >>>>>>
> >>>>>> We should discuss whether we agree on the theme of the first
> >>>>>> milestone (mentioned above). Given that, then, we should discuss
> what makes
> >>>>>> it and what doesn't (basically the above 2 lists).
> >>>>>>
> >>>>>> We also should discuss the rough time frame to complete this.
This
> >>>>>> email is to open the discussion. Your thoughts are welcome.
Thanks!
> >>>>>>
> >>>>>> Sangjin
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message