hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
Date Thu, 29 Oct 2015 22:51:27 GMT

    [ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981505#comment-14981505

Li Lu commented on YARN-4265:

Another key addition in this patch, compare to YARN-3942, is a "cache id plugin" mechanism.
One general problem the plugin needs to solve is "which exact storage should I look at on
a read request like ...". The user can add a plugin (for example, a Tez plugin) in the configuration.
At run time, on each read request, the plugin can help deciding if we need to look at the
summary storage only or the detailed (entity) storage. In this way, the whole storage decision
process of ATS 1.5 fine-grain cache storage will be:

- On writes, the application can decide if an entity should go to the summary storage or the
detailed storage. If it should go to the latter, the application need to provide a cache id
for this entity. 
- On reads, the application provides a plugin to decide if the look up can be performed on
the summery storage, or on the detailed storage. If it's the latter, the plugin provides a
cache id so that the storage should look at. 

> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>                 Key: YARN-4265
>                 URL: https://issues.apache.org/jira/browse/YARN-4265
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-4265-trunk.poc_001.patch
> To support the newly proposed APIs in YARN-4234, we need to create a new plugin timeline
store. The store may have similar behavior as the EntityFileTimelineStore proposed in YARN-3942,
but cache date in cache id granularity, instead of application id granularity. Let's have
this storage as a standalone one, instead of updating EntityFileTimelineStore, to keep the
existing store (EntityFileTimelineStore) stable. 

This message was sent by Atlassian JIRA

View raw message