hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
Date Tue, 06 May 2014 19:42:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991038#comment-13991038
] 

Chris Nauroth commented on HDFS-6326:
-------------------------------------

Daryn, thank you for taking a look at this.

bq. It's probably more appropriate for WebHdfsFileSystem to detect if acls are supported upon
the first call. It keeps the ugliness out of FsShell. It would be great to also push the hdfs-specific
exception handling into DistributedFileSystem.

I agree this would look cleaner, but then I would worry about a long-lived client process
recording that ACLs aren't supported, then a NameNode gets upgraded/reconfigured to support
ACLs, but the client continues to think it's unsupported.  This isn't a problem for the shell,
because the process is short-lived.  Maybe allow for a periodic retry to check for ACL support
again?

bq. Given that we now have extensible protobufs and json, is there any reason why FileStatus,
or FSPermission, etc don't return a boolean if there's an acl on the file? Then an acl call
is may only be made when necessary.

I discussed this in my first comment on this issue.  Here are some more details.

Regarding {{FileStatus}}, the same question came up on MAPREDUCE-5809 today.  Here is a copy-paste
of my response:

We considered adding the ACLs to {{FileStatus}}, but this would have been a backwards-incompatible
change. {{FileStatus}} implements {{Writable}} serialization, which is more brittle to version
compared to something like protobuf. {{FileStatus#write}} doesn't embed any kind of version
number, so there is no reliable way to tell at runtime if we are deserializing a pre-ACLs
{{FileStatus}} or a post-ACLs {{FileStatus}}. This would have had a high risk of breaking
downstream code or mixed versions that had used the {{Writable}} serialization. An alternative
would have been to skip serializing ACLs in {{FileStatus#write}}, but then there would have
been a risk of NPE for clients expecting a fully serialized object. This is discussed further
in the HDFS-4685 design doc on page 12:

https://issues.apache.org/jira/secure/attachment/12627729/HDFS-ACLs-Design-3.pdf

Regarding {{FsPermission}}, at one point the HDFS-4685 feature branch was using a previously
unused bit in the {{FsPermssion}} short as an ACL indicator.  This got pushback though on
HDFS-5923, and the ACL bit changes were backed out completely.  Here is a relevant comment:

https://issues.apache.org/jira/browse/HDFS-5923?focusedCommentId=13898370&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13898370

If you look at my attached HDFS-6326.1.patch, it reintroduces the ACL bit as a transient field,
toggled on the outbound {{FileStatus}} responses from NameNode, but not persisted to fsimage.
 Making it transient addresses some of the objections in HDFS-5923, but still leaves the specific
objections mentioned in the linked comment.

[~daryn] and [~wheat9], the two of you have been most involved in the debate on this, so would
you please review this comment, reconsider the trade-offs, and post your current opinion?
 At this point, considering all of the problems, I'm in favor of reintroducing the ACL bit
(patch v1).  The one problem with this approach is that if a 2.4.1 shell talks to a 2.4.0
NameNode, then it wouldn't display the '+', because the 2.4.0 NameNode wouldn't be toggling
the ACL bit.  I think it's worth accepting this relatively minor bug for one release to get
us past the larger issues.

> WebHdfs ACL compatibility is broken
> -----------------------------------
>
>                 Key: HDFS-6326
>                 URL: https://issues.apache.org/jira/browse/HDFS-6326
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 3.0.0, 2.4.0
>            Reporter: Daryn Sharp
>            Assignee: Chris Nauroth
>            Priority: Blocker
>         Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch
>
>
> 2.4 ACL support is completely incompatible with <2.4 webhdfs servers.  The NN throws
an {{IllegalArgumentException}} exception.
> {code}
> hadoop fs -ls webhdfs://nn/
> Found 21 items
> ls: Invalid value for webhdfs parameter "op": No enum constant org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
> [... 20 more times...]
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message