Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Fri, 13 Nov 2015 00:02:11 +0000 (UTC)
From: "Li Lu (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12904994.1444855725000.56914.1447372931072@Atlassian.JIRA>
In-Reply-To: <JIRA.12904994.1444855725000@Atlassian.JIRA>
References: <JIRA.12904994.1444855725000@Atlassian.JIRA>
 <JIRA.12904994.1444855725481@arcas>
Subject: [jira] [Updated] (YARN-4265) Provide new timeline plugin storage to
 support fine-grained entity caching
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Li Lu updated YARN-4265:
------------------------
    Attachment: YARN-4265.YARN-4234.001.patch

I refactored my code and added some unit tests. The current patch only depends on YARN-4234. I addressed comments raised by [~jlowe] in the previous round of review, with two pending actions that I think we can address in separate JIRAs:
# Build a "fall-back" plugin to have the same behavior of YARN-3942. 
# Make caching storage pluggable. 

For the second item, right now my patch does _not_ depend on YARN-4219. So I'm setting the v1.5 plugin storage to use the memory storage system as the "caching" storage, similar to YARN-3942.

I'm adding the v1.5 plugin storage (EntityGroupFSTimelineStore) in a module called hadoop-yarn-server-timeline-pluginstorage. This is slightly different to YARN-3942. We need a separate module because the new v1.5 storage depends on yarn-client, but we don't want ats server depends on yarn-client (The v1.5 storage is not a purely server side storage.). I'm naming it as "pluginstorage" because I'm considering to put the leveldb caching storage into this extension as well. 

> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>
>                 Key: YARN-4265
>                 URL: https://issues.apache.org/jira/browse/YARN-4265
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-4265-trunk.poc_001.patch, YARN-4265.YARN-4234.001.patch
>
>
> To support the newly proposed APIs in YARN-4234, we need to create a new plugin timeline store. The store may have similar behavior as the EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id granularity, instead of application id granularity. Let's have this storage as a standalone one, instead of updating EntityFileTimelineStore, to keep the existing store (EntityFileTimelineStore) stable. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)