hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2033) Investigate merging generic-history into the Timeline Store
Date Thu, 14 Aug 2014 05:31:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096592#comment-14096592

Zhijie Shen commented on YARN-2033:

[~djp], thanks for raising this question explicitly. Here're two points I'd like to highlight
for this work:

1. This patch doesn't intend to remove the existing FS based history store, but deprecate
it by removing the default configs about loading FS based history store. On the other hand,
the patch adds the history store that rides the timeline store, and use it as the default.
Given the user who is the early adopter of the generic history service wants to continue with
FS based history store, he needs to set the old configs explicitly (actually he should have
done it because by default the generic history service is not enabled), and the new generic
history service is still going to horner old configs for backward compatibility.

2. Though the generic history service (previously we call it application history server) is
introduced to Hadoop since 2.4, but it is not production ready. We have explicitly highlighted
it in the [documentation|http://hadoop.apache.org/docs/r2.4.0/hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Current_Status].
I agree it seems to be a bit aggressive to move from FS based history store to timeline store
based one as the default, however, I'm afraid it's the best choice at the current stage, because
FS based history store has several critical limitations: no caching, no retention, not scalable
and not supporting the secure mode. Unless we're able to solve all these problems (obviously
we don't have the bandwidth to do it now), it's risky to use FS based history store as the
default, in particular when the timeline server is going to be production ready. On the other
side, the aforementioned limitations have already been addressed by the timeline store (scalability
will be ensured by HBase timeline store). Hence timeline store based history store should
be a more reasonable and reliable default of new users.

> Investigate merging generic-history into the Timeline Store
> -----------------------------------------------------------
>                 Key: YARN-2033
>                 URL: https://issues.apache.org/jira/browse/YARN-2033
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Zhijie Shen
>         Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, YARN-2033.1.patch,
YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, YARN-2033.5.patch, YARN-2033.Prototype.patch,
YARN-2033_ALL.1.patch, YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
> Having two different stores isn't amicable to generic insights on what's happening with
applications. This is to investigate porting generic-history into the Timeline Store.
> One goal is to try and retain most of the client side interfaces as close to what we
have today.

This message was sent by Atlassian JIRA

View raw message