hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3187) Quotas for name space management
Date Thu, 29 May 2008 02:14:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600664#action_12600664
] 

Konstantin Shvachko commented on HADOOP-3187:
---------------------------------------------

# DFSAdmin.printHelp()
-- summary should be consistent with the detailed usage messages
<code>
      "\t[-refreshNodes] [-setQuota <dirname> <N>] [-clrQuota <dirname>]\n"
+
</code>
-- the same in printUsage()
-- The documentation for setQuota defines parameters in the reverse order -setQuota N name.
# I am not sure, but my first thought was that QuotaExceededException is a partial case of
AccessControlException.
One way to treat it is: if quota is exceeded for a directory then the write access to the
directory is forbidden.
# File QuotaExceededException.java is missing in the patch.
# FSNamesystem.clearQuota() should not be public.
# Please use **-comments for new methods (private or not) so that they appear in JavaDoc.
# unprotectedRenameTo() catches IOException which is never thrown.
# INodeDirectory.replaceChildWithSameName(newNode, oldNode) probably does not need 2 parameters.
You can just use newNode and search for the old inode using the new one's name.
You do a binary search anyway. And the method name (ChildWithSameName) reflects the semantics.
# INodeDirectoryWithQuota.count is int. This in fact imposes a restriction on the number of
files in hdfs compared to current implementation.
The restriction is imposed by making rootDir a INodeDirectoryWithQuota. The quota is not set
for the root by default,
but the count is an integer.
Right now our name-node cannot support Integer.MAX_VALUE number of files, but may be in the
future it will.
I think it is ok to have int count for directories that have their quotas set, and I think
it is all right to restrict 
the quota values (and therefore the count) by Integer.MAX_VALUE. But regular directories and
especially the root
should be able to have more entries.
# The same with ContentSummary. fileCount and directoryCount fields should remain long imo,
while quota can be int.
# This is not introduced by your patch but could you please remove redundant imports of java.io.DataInput
and DataOutput
# When creating files and directories you first increment counters for all directories with
quotas then actually create
a new directory entry at the specified path. You restore the counters for the directories
only if one of the quotas 
on the path happens to exceed the high mark. If something happens before file or directory
is actually
added but after the counters are incremented the quotas will be wrong. I think it would be
safer to
-# verify that no quotas are exceeded
-# add new directory entry
-# increment counters for directories with quotas on the path.
This will require splitting the updateCount() into 2 methods: one will verify quotas and and
the other will increment the counters.


> Quotas for name space management
> --------------------------------
>
>                 Key: HADOOP-3187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3187
>             Project: Hadoop Core
>          Issue Type: New Feature
>            Reporter: Robert Chansler
>            Assignee: Hairong Kuang
>         Attachments: quota.patch, QuotasDesign.pdf, QuotasDesign1.pdf
>
>
> Create a quota mechanism for name space management. 
> Quota set at a directory by super user restricts that number of names in and below that
directory.
> Quota tested by create() and rename().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message