hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Tucker (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11557) Empty directories may be recursively deleted without being listable
Date Tue, 28 Mar 2017 22:02:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946049#comment-15946049
] 

David Tucker edited comment on HDFS-11557 at 3/28/17 10:02 PM:
---------------------------------------------------------------

Indeed, I am able to reproduce with an internal Snakebite-like client and with the regular
client:

{code}
>>> import pydoofus
>>> super = pydoofus.namenode.v9.Client('namenode', 8020, auth={'effective_user':
'hdfs'})
>>> client = pydoofus.namenode.v9.Client('namenode', 8020, auth={'effective_user':
'nobody'})
>>> super.mkdirs('/test', 0777)
True
>>> client.mkdirs('/test/empty', 0222)
True
>>> client.get_listing('/test/empty')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/pydoofus/namenode/v9.py", line 666, in get_listing
    self.invoke('getListing', request, response)
  File "/usr/lib/python2.7/site-packages/pydoofus/namenode/v9.py", line 490, in invoke
    blob = self.channel.receive()
  File "/usr/lib/python2.7/site-packages/pydoofus/namenode/v9.py", line 310, in receive
    raise exceptions.create_exception(err_type, err_msg, call_id, err_code)
pydoofus.exceptions.AccessControlException: ERROR_APPLICATION: Permission denied: user=nobody,
access=READ_EXECUTE, inode="/test/empty":nobody:supergroup:d-w--w--w-
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1728)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1712)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1686)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:76)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4486)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:999)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:634)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

>>> client.delete('/test/empty', can_recurse=True)
True
{code}

{code}
[hdfs@rketcherside-hdponlynext-1 ~]$ hdfs dfs -mkdir /test
[hdfs@rketcherside-hdponlynext-1 ~]$ hdfs dfs -chmod 777 /test
[hdfs@rketcherside-hdponlynext-1 ~]$ hdfs dfs -ls /
Found 9 items
drwxrwxrwx   - yarn   hadoop          0 2017-03-27 10:20 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2017-03-27 10:20 /apps
drwxr-xr-x   - yarn   hadoop          0 2017-03-27 10:20 /ats
drwxr-xr-x   - hdfs   hdfs            0 2017-03-27 10:20 /hdp
drwxr-xr-x   - mapred hdfs            0 2017-03-27 10:20 /mapred
drwxrwxrwx   - mapred hadoop          0 2017-03-27 10:20 /mr-history
drwxrwxrwx   - hdfs   hdfs            0 2017-03-28 14:55 /test
drwxrwxrwx   - hdfs   hdfs            0 2017-03-28 09:21 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2017-03-28 09:21 /user

[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -mkdir /test/empty
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -chmod 222 /test/empty
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -ls /test/empty
ls: Permission denied: user=ambari-qa, access=READ_EXECUTE, inode="/test/empty":ambari-qa:hdfs:d-w--w--w-
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -rm -r /test/empty
17/03/28 14:57:45 INFO fs.TrashPolicyDefault: Moved: 'hdfs://rketcherside-hdponlynext-1.west.isilon.com:8020/test/empty'
to trash at: hdfs://rketcherside-hdponlynext-1.west.isilon.com:8020/user/ambari-qa/.Trash/Current/test/empty
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -ls /test
[ambari-qa@rketcherside-hdponlynext-1 ~]$ 
{code}


was (Author: dmtucker):
Indeed, I am able to reproduce with an internal Snakebite-like client and with the regular
client:

{code:python}
>>> import pydoofus
>>> super = pydoofus.namenode.v9.Client('namenode', 8020, auth={'effective_user':
'hdfs'})
>>> client = pydoofus.namenode.v9.Client('namenode', 8020, auth={'effective_user':
'nobody'})
>>> super.mkdirs('/test', 0777)
True
>>> client.mkdirs('/test/empty', 0222)
True
>>> client.get_listing('/test/empty')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/pydoofus/namenode/v9.py", line 666, in get_listing
    self.invoke('getListing', request, response)
  File "/usr/lib/python2.7/site-packages/pydoofus/namenode/v9.py", line 490, in invoke
    blob = self.channel.receive()
  File "/usr/lib/python2.7/site-packages/pydoofus/namenode/v9.py", line 310, in receive
    raise exceptions.create_exception(err_type, err_msg, call_id, err_code)
pydoofus.exceptions.AccessControlException: ERROR_APPLICATION: Permission denied: user=nobody,
access=READ_EXECUTE, inode="/test/empty":nobody:supergroup:d-w--w--w-
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1728)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1712)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1686)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:76)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4486)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:999)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:634)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

>>> client.delete('/test/empty', can_recurse=True)
True
{code}

{code}
[hdfs@rketcherside-hdponlynext-1 ~]$ hdfs dfs -mkdir /test
[hdfs@rketcherside-hdponlynext-1 ~]$ hdfs dfs -chmod 777 /test
[hdfs@rketcherside-hdponlynext-1 ~]$ hdfs dfs -ls /
Found 9 items
drwxrwxrwx   - yarn   hadoop          0 2017-03-27 10:20 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2017-03-27 10:20 /apps
drwxr-xr-x   - yarn   hadoop          0 2017-03-27 10:20 /ats
drwxr-xr-x   - hdfs   hdfs            0 2017-03-27 10:20 /hdp
drwxr-xr-x   - mapred hdfs            0 2017-03-27 10:20 /mapred
drwxrwxrwx   - mapred hadoop          0 2017-03-27 10:20 /mr-history
drwxrwxrwx   - hdfs   hdfs            0 2017-03-28 14:55 /test
drwxrwxrwx   - hdfs   hdfs            0 2017-03-28 09:21 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2017-03-28 09:21 /user

[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -mkdir /test/empty
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -chmod 222 /test/empty
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -ls /test/empty
ls: Permission denied: user=ambari-qa, access=READ_EXECUTE, inode="/test/empty":ambari-qa:hdfs:d-w--w--w-
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -rm -r /test/empty
17/03/28 14:57:45 INFO fs.TrashPolicyDefault: Moved: 'hdfs://rketcherside-hdponlynext-1.west.isilon.com:8020/test/empty'
to trash at: hdfs://rketcherside-hdponlynext-1.west.isilon.com:8020/user/ambari-qa/.Trash/Current/test/empty
[ambari-qa@rketcherside-hdponlynext-1 ~]$ hdfs dfs -ls /test
[ambari-qa@rketcherside-hdponlynext-1 ~]$ 
{code}

> Empty directories may be recursively deleted without being listable
> -------------------------------------------------------------------
>
>                 Key: HDFS-11557
>                 URL: https://issues.apache.org/jira/browse/HDFS-11557
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 2.7.3
>            Reporter: David Tucker
>            Assignee: Chen Liang
>
> To reproduce, create a directory without read and/or execute permissions (i.e. 0666,
0333, or 0222), then call delete on it with can_recurse=True. Note that the delete succeeds
even though the client is unable to check for emptiness and, therefore, cannot otherwise know
that any/all children are deletable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message