hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3942) Timeline store to read events from HDFS
Date Tue, 28 Jul 2015 14:01:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644394#comment-14644394

Jason Lowe commented on YARN-3942:

bq.  if you'd like to elaborate the drawback a bit, it will be helpful.

Yes the drawback is that one cannot do cross-application queries unless the entity type is
stored in the main database (i.e.: listed in yarn.timeline-service.entity-file-store.summary-entity-types).
 Another case that wouldn't work is where the query has multiple application IDs in it --
the query processing will choose the HDFS store of one of the applications and fail to find
entities for the others.  In practice the Tez UI only does cross-application queries on the
"All DAGs" front page, and that only needs a small amount of entity types as I listed above.
 Since that's the main use case we're optimizing for with this approach, it allows us to offload
most of the entity types from the leveldb database and serve them directly from HDFS.

So this solution is a tradeoff.  It limits the types of queries that the timeline server can
properly answer but significantly scales the single-node timeline server and decouples the
jobs posting events from the timeline server.  The latter is particularly interesting for
us, as we no longer have a mission-critical single node for running jobs that need to post
timeline events.

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure to mitigate
some of the issues we've seen with ATS v1 while waiting for ATS v2.  The intent of this plugin
is to provide a workable solution for running the Tez UI against the timeline server on a
large-scale clusters running many thousands of jobs per day.

This message was sent by Atlassian JIRA

View raw message