hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
Date Wed, 11 Sep 2013 20:10:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764707#comment-13764707
] 

Alejandro Abdelnur commented on HDFS-5122:
------------------------------------------

I would take a slightly different approach for this:

The WebHdfsFileSystem#getHttpUrlConnection() method is used for every FS call. In this method,
if the URI is for the NN (metadata operation, data transfers will be to a DN) do the following:

1* the standby NM should have a redirector HTTP servlet that bounces all HTTP calls from the
standby to the active.
2* Use the same utility classes DistributedFileSystem class uses to determine if hostname
in the URI is a logical name or not.
3* If the hostname is no a logical name, do current logic.
4* If the hostname is a logical name, resolve to any of the NN hosts, do a cheap FS call using
the chosen hostname.
4.1**If it works cache the chosen hostname and use it for all subsequent FS operations while
successful. 
4.2** If the call returns a redirect (automatic redirects are disabled) means you hit the
standby, select the other hostname and use it for all subsequent FS operations while successful.

4.3** if the call returns a cannot connect or error means you hit a NN that is 'dead', fallback
to the other NN hostname and use it for all subsequent FS operations while successful. 
5* When a subsequent URL call fails do #4.3
6* Make sure you have logic to avoid infinite loop  of bouncing between NNs in case both are
dead.
7*The WebHDFS delegation token service should use similar logic like the DFS delegation token
to convert from logical name to hostname the service in the token.

NOTE: I'm not familiar on WebHDFSFileSystem current retry logic, and some of this could be
already take can of it.
                
> WebHDFS should support logical service names in URIs
> ----------------------------------------------------
>
>                 Key: HDFS-5122
>                 URL: https://issues.apache.org/jira/browse/HDFS-5122
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha, webhdfs
>    Affects Versions: 2.1.0-beta
>            Reporter: Arpit Gupta
>            Assignee: Haohui Mai
>         Attachments: HDFS-5122.patch
>
>
> For example if the dfs.nameservices is set to arpit
> {code}
> hdfs dfs -ls webhdfs://arpit:50070/tmp
> or 
> hdfs dfs -ls webhdfs://arpit/tmp
> {code}
> does not work
> You have to provide the exact active namenode hostname. On an HA cluster using dfs client
one should not need to provide the active nn hostname

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message