hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
Date Mon, 12 May 2014 17:26:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995278#comment-13995278
] 

Haohui Mai commented on HDFS-6326:
----------------------------------

bq. the assumption in the web UI code is completely wrong and prevents any future use of any
upper bits in the mask.

The point is to demonstrate how other clients of webhdfs can potentially depend on this behavior.
Though the bug is within our reach and can be fixed by ourselves, the webhdfs protocol is
intended to be implemented by 3rd-party clients which are totally out of our control. I have
no problem to declare this is a wrong assumption but the sad fact is that 3rd-party clients
might depend on it.

bq. I think avoiding the performance penalty of double rpc/http calls just to print a "+"
is more important than working around a web bug.

Having the bit obviously makes implementing ls / web ui much easier, but it comes with a performance
cost on the critical path. We've implemented this approach before and reverted it to the current
state, as Chris mentioned in earlier comments.

{{ls}} is not in the critical path, but adding a new field into the {{FileStatus}} response
is. Provided that the {{FileStatus}} field is relatively short (~200 bytes), adding a field
for ACL increases the network traffic for listing files by 5%. The overhead is indeed significant
as listing files is the first step of distcp which is done within a single thread. We've seen
use cases that this step can take more than 7 hours.

Again having an ACL bit makes things easier to implement, but failing to justify the performance
concerns leads Chris and I to decide to keep ACL out of the critical path, and to move the
complexity to ls / web UI.
 
Maybe I don't understand what you're proposing, can you propose your changes more concretely,
just like what Chris did in  https://issues.apache.org/jira/browse/HDFS-6326?focusedCommentId=13991788&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13991788

> WebHdfs ACL compatibility is broken
> -----------------------------------
>
>                 Key: HDFS-6326
>                 URL: https://issues.apache.org/jira/browse/HDFS-6326
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 3.0.0, 2.4.0
>            Reporter: Daryn Sharp
>            Assignee: Chris Nauroth
>            Priority: Blocker
>         Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch
>
>
> 2.4 ACL support is completely incompatible with <2.4 webhdfs servers.  The NN throws
an {{IllegalArgumentException}} exception.
> {code}
> hadoop fs -ls webhdfs://nn/
> Found 21 items
> ls: Invalid value for webhdfs parameter "op": No enum constant org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
> [... 20 more times...]
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message