hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
Date Sat, 01 Feb 2014 21:04:09 GMT

    [ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888738#comment-13888738

Vinod Kumar Vavilapalli commented on YARN-1530:

Thanks for the comments everyone. Some responses follow:

bq. Yes, proxy server inside library, but only in AM not containers.
Yes, that's a good idea and inline with what I might have hinted in the doc. When containers
also start writing out events, we need to have these intermediate aggregators for scalability.
I was thinking NodeManagers but your idea of AMs is better. If we make everyone only depend
on a library, the underlying protocol can be either RPC or REST together with a RPC server
or a proxy like you mentioned. If REST is itself the first-class API, yes, proxy-server is
the way.

bq. I am also a bit nervous about using the history service for recovery or as a backend for
the current MR APIs if we have a pub/sub system as a link between the applications and the
history service.
Agreed. This is a problem that exists even today with MR apps in a way with HDFS acting as
a pubsub system. We've seen the corner cases you are hinting at even with the HDFS based history
files. I think we now understand enough about these edge cases. I haven't yet made the jump
to making these events be the base for recovery, but points noted for when we wish to.

bq. 1. Is the expectation that people will be able to use this for INFO, WARN, ERROR type
application logging?
Application logging is not supposed to be through this channel. This feature is fundamentally
for meta-event information - frankly, the kind we log in MR JobHistory files. We already have
the ability for containers to write logs to a local log-directory and the corresponding features
for aggregation, rendering. This is a high throughput path that IMO cannot be sustained by
the solution of this JIRA. I'll make it clear in the doc.
bq. 2. Regarding app-specific UIs, is it going to be possible to embed app-specific UIs with
YARN's UI, instead of having to run an app-specific web-ui? There is some mention of JS UIs,
but it's a little unclear whether this would be embedded in YARN, or served from "somewhere"
(to quote the docs. If it's served from the RM (or some other web-ui in YARN), will it be
up to ops to decide which libraries are embedded, or up to
I thought I wrote it clearly, will have to reread and edit if necessary. The idea is to either
let users host their UI's elsewhere or give an ability for admins to install UIs (that they
have vetted for stability/security etc) in the HIstoryServer itself.
bq. 3. What are the planned "out of the box implementations" for both the storage and transport
layers? REST+LevelDB+HBase? Are Flume and Kafka implementations expected to happen outside
of the YARN project?
We are planning Library(wrapping REST)+LevelDB+LocalFS for smaller deployments and out of
the box. We will then move to working on Library(wrapping REST)+LevelDB+HBase for larger deployments.

The discussion of Flume/Kafka is related to the event-aggregation which we currently are doing
via a simple REST put to a web-service. Whether/when we move to those implementations will
depend on the urgency with which we run into issues mentioned in the doc with REST APIs. But
it's open at this point of time.

> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>                 Key: YARN-1530
>                 URL: https://issues.apache.org/jira/browse/YARN-1530
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>         Attachments: application timeline design-20140108.pdf, application timeline design-20140116.pdf,
application timeline design-20140130.pdf
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework data all by
itself as YARN doesn't have a common solution. This JIRA attempts to solve the storage, management
and serving of per-framework data from various applications, both running and finished. The
aim is to change YARN to collect and store data in a generic manner with plugin points for
frameworks to do their own thing w.r.t interpretation and serving.

This message was sent by Atlassian JIRA

View raw message