hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
Date Wed, 26 Oct 2011 20:19:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136305#comment-13136305
] 

Daryn Sharp commented on HDFS-2450:
-----------------------------------

# "can you get thatScheme before making if ..."
#* Gladly!  That was one of the aforementioned things I was "itching" to do, but removed before
posting the patch to minimize the change.
# "All checks seem to be using defaultPort"
#* No, the default port is only used if there is not a port in the uri
# "Why remove?"
#* Default uri is sufficient because {{FileSystem.get(defaultUri, conf).getUri()}} equals
{{defaultUri}}.  Also note about instantiating the default fs:
#*# It's relatively expensive, and horrifically bad if the fs cache is disabled to create
a new fs
#*# If the default fs is down, the thread blocks while rpc retries occur
#*# The thrown {{RuntimeException}} is likely blow up the client, as opposed to the {{IllegalArgumentException}}
the caller expects.
# "If thisAuthority and thatAuthority is null, the new mehod throw IllegalArgumentException"
#* No, the == on the authorities will catch this case.
# "If thisAuthority and thatAuthority are not null, you end up parsing the URI for thatAuthority
twice."
#* Yes, once before canonicalization, once after canonicalization.
# "DistributedFileSystem#makeQualified remove makes me concerned"
#* The only reason it existed is because {{NameNode.getUri()}} was stripping off the default
port.  This alone would cause {{p = new Path("hdfs://host:default-port/file"); fs = p.getFileSystem(conf)}}
to be incompatible because the path has a port, but the fs stripped it off.  So an override
of {{checkPath}} and {{makeQualified}} were added to strip the default port from paths too.
 All of it is completely unnecessary, especially with canonicalization.

Tests:
# full test suite
# manual testing with FsShell "ls":
#* scheme://short
#* scheme://short:port
#* scheme://short.partial
#* scheme://short.partial:port
#* scheme://fqdn
#* scheme://fqdn:port
#* scheme://ip
#* scheme://ip:port
# adding more unit tests to automate stressing {{checkPath}} via {{DFS}}, both positive and
negative cases
                
> Only complete hostname is supported to access data via hdfs://
> --------------------------------------------------------------
>
>                 Key: HDFS-2450
>                 URL: https://issues.apache.org/jira/browse/HDFS-2450
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.205.0
>            Reporter: Rajit Saha
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450.patch, IP vs. Hostname.pdf
>
>
> If my complete hostname is  host1.abc.xyz.com, only complete hostname must be used to
access data via hdfs://
> I am running following in .20.205 Client to get data from .20.205 NN (host1)
> $hadoop dfs -copyFromLocal /etc/passwd  hdfs://host1/tmp
> copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal <localsrc> ... <dst>]
> $hadoop dfs -copyFromLocal /etc/passwd  hdfs://host1.abc/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal <localsrc> ... <dst>]
> $hadoop dfs -copyFromLocal /etc/passwd  hftp://host1.abc.xyz/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal <localsrc> ... <dst>]
> Only following is supported 
> $hadoop dfs -copyFromLocal /etc/passwd  hdfs://host1.abc.xyz.com/tmp/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message