hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
Date Wed, 11 Sep 2013 20:10:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764707#comment-13764707

Alejandro Abdelnur commented on HDFS-5122:

I would take a slightly different approach for this:

The WebHdfsFileSystem#getHttpUrlConnection() method is used for every FS call. In this method,
if the URI is for the NN (metadata operation, data transfers will be to a DN) do the following:

1* the standby NM should have a redirector HTTP servlet that bounces all HTTP calls from the
standby to the active.
2* Use the same utility classes DistributedFileSystem class uses to determine if hostname
in the URI is a logical name or not.
3* If the hostname is no a logical name, do current logic.
4* If the hostname is a logical name, resolve to any of the NN hosts, do a cheap FS call using
the chosen hostname.
4.1**If it works cache the chosen hostname and use it for all subsequent FS operations while
4.2** If the call returns a redirect (automatic redirects are disabled) means you hit the
standby, select the other hostname and use it for all subsequent FS operations while successful.

4.3** if the call returns a cannot connect or error means you hit a NN that is 'dead', fallback
to the other NN hostname and use it for all subsequent FS operations while successful. 
5* When a subsequent URL call fails do #4.3
6* Make sure you have logic to avoid infinite loop  of bouncing between NNs in case both are
7*The WebHDFS delegation token service should use similar logic like the DFS delegation token
to convert from logical name to hostname the service in the token.

NOTE: I'm not familiar on WebHDFSFileSystem current retry logic, and some of this could be
already take can of it.
> WebHDFS should support logical service names in URIs
> ----------------------------------------------------
>                 Key: HDFS-5122
>                 URL: https://issues.apache.org/jira/browse/HDFS-5122
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha, webhdfs
>    Affects Versions: 2.1.0-beta
>            Reporter: Arpit Gupta
>            Assignee: Haohui Mai
>         Attachments: HDFS-5122.patch
> For example if the dfs.nameservices is set to arpit
> {code}
> hdfs dfs -ls webhdfs://arpit:50070/tmp
> or 
> hdfs dfs -ls webhdfs://arpit/tmp
> {code}
> does not work
> You have to provide the exact active namenode hostname. On an HA cluster using dfs client
one should not need to provide the active nn hostname

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message