hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Saxena <vsaxena.va...@gmail.com>
Subject Re: [YARN-2928] first drop on trunk
Date Fri, 13 Nov 2015 20:16:09 GMT
The list
https://issues.apache.org/jira/issues/?jql=labels%20%3D%20yarn-2928-1st-milestone
looks good to me.

Regards,
Varun Saxena.


On Sat, Nov 14, 2015 at 12:09 AM, Sangjin Lee <sjlee@apache.org> wrote:

> I took a stab at labeling JIRAs that I think we need for the first
> milestone drop, and came up with 17 JIRAs:
> https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
>
> Please take a look at them, and comment on them.
>
> Thanks,
> Sangjin
>
> On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <joep@twitter.com> wrote:
>
>> Agreed with the theme and lists for the merge back of the first
>> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
>> suggestion with the two lists Sangjin.
>>
>> As we discussed in the weekly sync with the ATS v2 team, right after
>> we've merged back, the next thing to decide is what the story will be to
>> run both ATS v1 and ATS v2 in parallel so that we can start discussing when
>> we can do a merge to branch-2.
>> We'll want people (including ourselves) to be able to run ATS v2 on a
>> trial basis w/o impacting existing use-cases on a branch-2 based release.
>> The merge-back to branch-2 will be a separate topic of discussion though.
>>
>> Thanks,
>>
>> Joep
>>
>>
>> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <kasha@cloudera.com>
>> wrote:
>>
>>> +1
>>>
>>> In favor of merging a version working end-to-end at alpha quality to
>>> trunk.
>>>
>>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sjlee@apache.org> wrote:
>>>
>>>> Just a reminder that I'd like input from folks that are working on this
>>>> feature (as well as anyone who's interested or course!).
>>>>
>>>> I propose creating a label to start labeling things we'd like to see
>>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
>>>>
>>>> Thanks,
>>>> Sangjin
>>>>
>>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vsaxena.varun@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Sangjin,
>>>>>
>>>>> Thanks for initiating the discussion.
>>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
>>>>> somewhat related.
>>>>>
>>>>> Regards,
>>>>> Varun Saxena.
>>>>>
>>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sjlee@apache.org>
wrote:
>>>>>
>>>>>> In the past several months, the timeline service v.2 team has made
a
>>>>>> tremendous progress. We have a working storage implementation based
on
>>>>>> HBase, timeline collectors, timeline readers with some of the more
>>>>>> important queries and filters, integration with RM and NM, distributed
>>>>>> shell and mapreduce, and some basic UI to boot. We're getting real
close to
>>>>>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>>>>>> here)!
>>>>>>
>>>>>> I think it is time to discuss defining a merge to trunk of an
>>>>>> alpha-quality release as our first milestone so that a wider audience
has a
>>>>>> chance to try it out. This doesn't replace the timeline service (ATS)
v.1
>>>>>> yet, but it would be a great chance to get feedback.
>>>>>>
>>>>>> I think the theme is essentially a basic but complete end-to-end
flow
>>>>>> that includes the write path and the read path and some UI. These
are the
>>>>>> key major things we may want to complete before we consider merging
the
>>>>>> first milestone:
>>>>>> - application aggregation (YARN-3816
>>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>>>>>> - flow run compaction work (YARN-4062
>>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>>>>>> - finalize the metrics storage (YARN-4053
>>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>>>>>> - improve queries and filters (YARN-3863
>>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>>>>>> - UI POC based on the new YARN UI framework (YARN-4097
>>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>>>>>
>>>>>> In addition to these, we would like to close a few more JIRAs that
>>>>>> we're currently working on. Also, in terms of the app integration,
we can
>>>>>> debate whether we stick to the distributed shell for now or spend
some more
>>>>>> effort to round out the mapreduce support.
>>>>>>
>>>>>> I also think the following major things are probably out of scope
for
>>>>>> this first drop:
>>>>>> - time-based (offline) user/queue aggregation based on Phoenix (
>>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>>>>>> - fault-tolerant storage (YARN-4061
>>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>>>>>> - timeline collector as a separate daemon (YARN-3033
>>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>>>>>> - timeline collector containerization
>>>>>> - compatibility with v.1 (YARN-3196
>>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>>>>>> - support for off-cluster timeline clients (YARN-3981
>>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>>>>>
>>>>>> We should discuss whether we agree on the theme of the first
>>>>>> milestone (mentioned above). Given that, then, we should discuss
what makes
>>>>>> it and what doesn't (basically the above 2 lists).
>>>>>>
>>>>>> We also should discuss the rough time frame to complete this. This
>>>>>> email is to open the discussion. Your thoughts are welcome. Thanks!
>>>>>>
>>>>>> Sangjin
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message