hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Lu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3942) Timeline store to read events from HDFS
Date Fri, 25 Sep 2015 17:50:05 GMT

     [ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Li Lu updated YARN-3942:
    Attachment: YARN-3942-leveldb.002.patch

Thanks [~gss2002]! I modified the patch with your changes. As a quick update, the current
Leveldb cache storage patch addresses the (potential) OOM issue. However, now we inevitably
have to load all entities as a whole when refreshing the cache, which will introduce longer
latency. I discussed this with [~hitesh] and [~xgong] and seems like the best solution to
the latency problem will be reducing the granularity of the caches. Instead of loading (and
caching) entities for a whole app, we need to load and cache a set of entities, like those
ones in the same Tez DAG. Let's focus on the current fix in this JIRA, and open a new JIRA
for the next step? Comments and suggestions are more than welcome. 

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942-leveldb.001.patch, YARN-3942-leveldb.002.patch, YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure to mitigate
some of the issues we've seen with ATS v1 while waiting for ATS v2.  The intent of this plugin
is to provide a workable solution for running the Tez UI against the timeline server on a
large-scale clusters running many thousands of jobs per day.

This message was sent by Atlassian JIRA

View raw message