hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Kambatla <ka...@cloudera.com>
Subject Re: [YARN-2928] first drop on trunk
Date Wed, 04 Nov 2015 20:18:30 GMT

In favor of merging a version working end-to-end at alpha quality to trunk.

On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sjlee@apache.org> wrote:

> Just a reminder that I'd like input from folks that are working on this
> feature (as well as anyone who's interested or course!).
> I propose creating a label to start labeling things we'd like to see done
> before the first milestone. How about "yarn-2928-1st-milestone"?
> Thanks,
> Sangjin
> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vsaxena.varun@gmail.com>
> wrote:
>> Hi Sangjin,
>> Thanks for initiating the discussion.
>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
>> somewhat related.
>> Regards,
>> Varun Saxena.
>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sjlee@apache.org> wrote:
>>> In the past several months, the timeline service v.2 team has made a
>>> tremendous progress. We have a working storage implementation based on
>>> HBase, timeline collectors, timeline readers with some of the more
>>> important queries and filters, integration with RM and NM, distributed
>>> shell and mapreduce, and some basic UI to boot. We're getting real close to
>>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>>> here)!
>>> I think it is time to discuss defining a merge to trunk of an
>>> alpha-quality release as our first milestone so that a wider audience has a
>>> chance to try it out. This doesn't replace the timeline service (ATS) v.1
>>> yet, but it would be a great chance to get feedback.
>>> I think the theme is essentially a basic but complete end-to-end flow
>>> that includes the write path and the read path and some UI. These are the
>>> key major things we may want to complete before we consider merging the
>>> first milestone:
>>> - application aggregation (YARN-3816
>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>>> - flow run compaction work (YARN-4062
>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>>> - finalize the metrics storage (YARN-4053
>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>>> - improve queries and filters (YARN-3863
>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>>> - UI POC based on the new YARN UI framework (YARN-4097
>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>> In addition to these, we would like to close a few more JIRAs that we're
>>> currently working on. Also, in terms of the app integration, we can debate
>>> whether we stick to the distributed shell for now or spend some more effort
>>> to round out the mapreduce support.
>>> I also think the following major things are probably out of scope for
>>> this first drop:
>>> - time-based (offline) user/queue aggregation based on Phoenix (
>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>>> - fault-tolerant storage (YARN-4061
>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>>> - timeline collector as a separate daemon (YARN-3033
>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>>> - timeline collector containerization
>>> - compatibility with v.1 (YARN-3196
>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>>> - support for off-cluster timeline clients (YARN-3981
>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>> We should discuss whether we agree on the theme of the first milestone
>>> (mentioned above). Given that, then, we should discuss what makes it and
>>> what doesn't (basically the above 2 lists).
>>> We also should discuss the rough time frame to complete this. This email
>>> is to open the discussion. Your thoughts are welcome. Thanks!
>>> Sangjin

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message