hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7933) Viewfs changes for MAPREDUCE-3529
Date Tue, 03 Jan 2012 19:58:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178986#comment-13178986

Siddharth Seth commented on HADOOP-7933:

bq. Minor: Maybe store mountPoints.get.target.targetFileSystem in a temp instead of invoking
Could be done - but i don't believe this lookup is very expensive.

bq. It appears Credentials was selected for convenience since that's what the MR's TokenCache
uses internally? It appears Credentials was selected for convenience since that's what the
MR's TokenCache uses internally?
The API is limited private (at least for now) to MR and HDFS - so I don't see this as a very
big issue. Changing it to a set is a simple change though. Thoughts ?

bq. Would it be reasonable to populate the Credentials/Set directly? Ie. ViewFileSystem takes
a set of existing tokens, then returns the subset of tokens for its mounts that are not already
in the existing tokens.
Was returning all tokens (instead of just new ones) to be consistent with the existing API.
Having this call populate tokens into the Credentials object was an option - but that increases
the cross-component dependency.

bq. I'm concerned that this appears to be creating a cross-component dependency whereby ViewFileSystem
it required to "assume" how MR's TokenCache will want the tokens keyed within its Credentials.
I'm not very happy with this either - suggestion on a better approach ? Again, being LimitedPrivate
to MR and HDFS limits the impact.

bq. I also noticed getDelegationTokens(renewer,creds) is returning null where getDelegationTokens(renewer)
would return a 0-length list.
I believe getDelegationTokens(renewer, creds) is also returning an empty list and not null.
However, there is a potential issue with the ViewFileSystem implementation - {{result = new
ArrayList<Token<?>>(initialListSize)}} - if there's multiple mount points - part
of the array will not be populated.

> Viewfs changes for MAPREDUCE-3529
> ---------------------------------
>                 Key: HADOOP-7933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7933
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>            Priority: Critical
>             Fix For: 0.23.1
>         Attachments: HADOOP-7933.txt, HADOOP7933_v1.txt, HADOOP7933_v2.txt, HDFS2665_v1.txt,
> ViewFs.getDelegationTokens returns a list of tokens for the associated namenodes. Credentials
serializes these tokens using the service name for the actual namenodes. Effectively, tokens
are not cached for viewfs (some more details in MR 3529). Affects any job which uses the TokenCache
in tasks along with viewfs (some Pig jobs).
> Talk to Jitendra about this, some options
> 1. Change Credentials.getAllTokens to return the key, instead of just a token list (associate
the viewfs canonical name with a token in credentials)
> 2. Have viewfs issue a fake token.
> Both of these would allow for a single viewfs configuration only.
> 3. An additional API in FileSystem - something like getDelegationTokens(String renewer,
Credentials credentials) - which would check the credentials object before making token requests
to the actual namenode.
> 4. An additional API in FileSystem - getCanonicalServiceNames - similar to getDelegationTokens,
which would return service names for the actual namenodes. TokenCache/Credentials can work
using this list.
> 5. have getDelegationTokens check the current UGI - and fetch tokens only if they don't
> Have a quick patch for 3, along with associated MR changes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message