hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9617) HA HDFS client is too strict with validating URI authorities
Date Mon, 03 Jun 2013 19:58:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13673489#comment-13673489
] 

Aaron T. Myers commented on HADOOP-9617:
----------------------------------------

bq. I think something else is wrong because I specifically made port/no-port work last I time
I was in that code. Canonicalizing the uri is supposed to handle it.

Sorry, not quite sure I follow. What else do you think is wrong?

bq. I believe it's incorrect behavior. Users should not have to fully qualify path authorities
with ports. It will break customers.

I think this pattern of using FileSystem with Paths with different authorities is itself incorrect.
One should not create a FileSystem object using one authority and then pass Paths to that
FileSystem which use another authority. Instead, users should be calling Path#getFileSystem
to get the proper FS associated with that Path.

Regardless, this issue is moot because the change was an inadvertent backward incompatible
one, which I think we all agree should be fixed.
                
> HA HDFS client is too strict with validating URI authorities
> ------------------------------------------------------------
>
>                 Key: HADOOP-9617
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9617
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, ha
>    Affects Versions: 2.0.5-alpha
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: HADOOP-9617.patch
>
>
> HADOOP-9150 changed the way FS URIs are handled to prevent attempted DNS resolution of
logical URIs. This has the side effect of changing the way Paths are verified when passed
to a FileSystem instance created with an authority that differs from the authority of the
Path. Previous to HADOOP-9150, a default port would be added to either authority in the event
that either URI did not have a port. Post HADOOP-9150, no default port is added. This means
that a FileSystem instance created using the URI "hdfs://ha-logical-uri:8020" will no longer
process paths containing just the authority "hdfs://ha-logical-uri", and will throw an error
like the following:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS: hdfs://ns1/user/hive/warehouse/sample_07/sample_07.csv,
expected: hdfs://ns1:8020
> 	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:625)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:173)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
> {noformat}
> Though this is not necessarily incorrect behavior, it is a backward-incompatible change
that at least breaks certain clients' ability to connect to an HA HDFS cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message