hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinglun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14633) The StorageType quota and consume in QuotaFeature is not handled when rename and setStoragePolicy etc.
Date Mon, 08 Jul 2019 13:12:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880323#comment-16880323

Jinglun commented on HDFS-14633:

Hi [~ayushtkn], thanks for your explanation of backward compatibility, I agree it will be
a problem.:) 
I read the code about setStoragePolicy's permission check, seems everyone with FsAction.WRITE
to the INode could change the policy. The super user priviledge is not checked. So not only
the admin but normal users can change it.
About rename, supposing I have a file /dir-1/A with storage policy HOT inherited from it's
parent. /dir-2 is a directory with storage policy COLD. After I rename /dir-1/A to /dir-2/A,
the consume in /dir-2 quota feature should be ARCHIVE, but it's not. The quota usage recorded
in /dir-2 doesn't match the count result of /dir-2. Also the ARCHIVE quota in /dir-2 is not
checked(It checks the SSD quota). I upload a test case to explain it.
About setStoragePolicy, it's not handled at all. I tried to fix setStoragePolicy when it's
called on a file in HDFS-14568, and it's not enough. Direcotry should also be handled.
The addBlock and delete rpcs handle the quota well, so I remove them from the jira description.
Sorry for the mistake, my bad.


Upload HDFS-14633-testcases-explanation to explain:
1. setStoragePolicy could be called by anyone with a write permission.
2. rename causes inconsistent in the count result and the usage recored in quota feature.


Looking forward to your comments. Thanks~


> The StorageType quota and consume in QuotaFeature is not handled when rename and setStoragePolicy
> -------------------------------------------------------------------------------------------------------
>                 Key: HDFS-14633
>                 URL: https://issues.apache.org/jira/browse/HDFS-14633
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Jinglun
>            Assignee: Jinglun
>            Priority: Major
> The NameNode manages the global state of the cluster. We should always take NameNode's
records as the sole criterion because no matter what inconsistent is the NameNode should finally
make everything right based on it's records. Let's call it rule NSC(NameNode is the Sole Criterion).
That means when we do all quota related rpcs, we do the quota check according to NameNode's
records regardless of any inconsistent situation, such as the replicas doesn't match the storage
policy of the file, or the replicas count doesn't match the file's set replica.
>  The work SPS deals with the wrongly palced replicas. There is a thought about putting
off the consume update of the DirectoryQuota until all replicas are re-placed by SPS. I can't
agree that because if we do so we will abandon letting the NameNode's records to be the sole
criterion. The block replication is a good example of the rule NSC. When we count the consume
of a file(CONTIGUOUS), we multiply the replication factor with the file's length, no matter
the blocks are under replicated or excessed. We should do the same thing for the storage type
>  Another concern is the change will let setStoragePolicy throw QuotaByStorageTypeExceededException
which it doesn't before. I don't think it's a big problem since the setStoragePolicy already
throws IOException. Or we can wrap the QuotaByStorageTypeExceededException in an IOException,
but I won't recommend that because it's ugly.
>  To make storage type consume follow the rule NSC, we need change rename(moving a file
with storage policy inherited from it's parent) and setStoragePolicy. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message