hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7967) Need generalized multi-token filesystem support
Date Tue, 07 Feb 2012 21:56:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202844#comment-13202844

Daryn Sharp commented on HADOOP-7967:

We should continue the discussion on MAPREDUCE-3825.  There is no dispute that multi-token
support exists.  However, it's broken in a number of ways, and is incompatible with a {{TokenCache}}
that assumes a 1 to 1 service/token mapping.

Regarding efficiency, leaking duplicate tokens everywhere increases the memory load on the
namenode, fills the edit logs, increases the txn burden, increases startup time, increases
GC time, leaked tokens aren't cancelled so they live until they expire, and it increases the
load on the secret manager.

It may be debated if duplicate tokens are a bug or performance, but fixing the duplicate tokens
is only incidentally fixed when fixing the bugs such as the broken viewfs mount table.
> Need generalized multi-token filesystem support
> -----------------------------------------------
>                 Key: HADOOP-7967
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7967
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.24.0, 0.23.1
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7967-2.patch, HADOOP-7967-3.patch, HADOOP-7967-4.patch, HADOOP-7967.patch
> Multi-token filesystem support and its interactions with the MR {{TokenCache}} is problematic.
 The {{TokenCache}} tries to assume it has the knowledge to know if the tokens for a filesystem
are available, which it can't possibly know for multi-token filesystems.  Filtered filesystems
are also problematic, such as har on viewfs.  When mergeFs is implemented, it too will become
a problem with the current implementation.  Currently {{FileSystem}} will leak tokens even
when some tokens are already present.
> The decision for token acquisition, and which tokens, should be pushed all the way down
into the {{FileSystem}} level.  The {{TokenCache}} should be ignorant and simply request tokens
from each {{FileSystem}}.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message