hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
Date Thu, 04 Dec 2014 23:44:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234801#comment-14234801
] 

Jian He commented on YARN-2837:
-------------------------------

Thanks Zhijie and Li, some comments on my side:

- Logic in {{Version loadedVersion = loadVersion();}} Consider this scenario: CURRENT_VERSION_INFO
= 2.0; there’s no version info currently saved in store-store. {{loadVersion}} returns 1.0;
It’ll throw inCompatible exception, even though it should not. 
- we can probably use protobuf to incorporate both tokenIdentifer and the renewDate to support
better compatibility. e.g. RMDelegationTokenIdentifierData
{code}
      renewDate = in.readLong();
      tokenId.readFields(in);
{code}
- {{LeveldbTimelineStateStore#updateToken}}, it’s always adding new token, the old token
still remain, we should remove the old token
- {{AbstractDelegationTokenSecretManager#delegationTokenSequenceNumber}} is not updated on
recovery; the implementation seems using sequenceNumber as the key, we need to keep track
of the latest sequenceNumber so that it can be recovered.
- the following looks ok , a simpler way might be to just concatenate two strings.
{code}
.add(TOKEN_MASTER_KEY_ENTRY_PREFIX).add(Integer.toString(keyId)){code}
- the following log is present in both {{TimelineDelegationTokenSecretManager#storeNewMasterKey}}
and the underlying state-store implementation; only printing  in one place is enough. similar
to other operations.
{code}
      if (LOG.isDebugEnabled()) {
        LOG.debug("Storing master key " + key.getKeyId());
      }
{code}
- FILENAME->DB_NAME; leveldb-state-store.ldb -> timeline-state-store.ldb
- default path for state store is the same as time-line store for application data. If apps
posts massive data in store, will that also affect system data seek performance ? If so, we
should have a different store path from the one for apps.  Or we could force user to configure
the path properly and throw exception otherwise.

> Timeline server needs to recover the timeline DT when restarting
> ----------------------------------------------------------------
>
>                 Key: YARN-2837
>                 URL: https://issues.apache.org/jira/browse/YARN-2837
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: timelineserver
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>            Priority: Blocker
>             Fix For: 2.7.0
>
>         Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch
>
>
> Timeline server needs to recover the stateful information when restarting as RM/NM/JHS
does now. So far the stateful information only includes the timeline DT. Without recovery,
the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more
after the timeline server is restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message