hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bc Wong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
Date Tue, 09 Sep 2014 19:02:32 GMT

    [ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127403#comment-14127403

bc Wong commented on YARN-1530:

bq. The current writing channel allows the data to be available on the timeline server immediately

Let's have reliability before speed. I think one of the requirement of ATS is: *The channel
for writing events should be reliable.*

I'm using *reliable* here in a strong sense, not the TCP-best-effort style reliability. HDFS
is reliable. Kafka is reliable. (They are also scalable and robust.) A normal RPC connection
is not. I don't want the ATS to be able to slow down my writes, and therefore, my applications,
at all. For example, an ATS failover shouldn't pause all my applications for N seconds. A
direct RPC to the ATS for writing seems a poor choice in general.

Yes, you could make a distributed reliable scalable "ATS service" to accept writing events.
But that seems a lot of work, while we can leverage existing technologies.

If the channel itself is pluggable, then we have lots of options. Kafka is a very good choice,
for sites that already deploy Kafka and know how to operate it. Using HDFS as a channel is
also a good default implementation, for people already know how to scale and manage HDFS.
Embedding a Kafka broker with each ATS daemon is also an option, if we're ok with that dependency.

> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>                 Key: YARN-1530
>                 URL: https://issues.apache.org/jira/browse/YARN-1530
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>         Attachments: ATS-Write-Pipeline-Design-Proposal.pdf, ATS-meet-up-8-28-2014-notes.pdf,
application timeline design-20140108.pdf, application timeline design-20140116.pdf, application
timeline design-20140130.pdf, application timeline design-20140210.pdf
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework data all by
itself as YARN doesn't have a common solution. This JIRA attempts to solve the storage, management
and serving of per-framework data from various applications, both running and finished. The
aim is to change YARN to collect and store data in a generic manner with plugin points for
frameworks to do their own thing w.r.t interpretation and serving.

This message was sent by Atlassian JIRA

View raw message