hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1751) Intrinsic limits for HDFS files, directories
Date Thu, 17 Mar 2011 23:33:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008216#comment-13008216
] 

Jakob Homan commented on HDFS-1751:
-----------------------------------

First off, this JIRA is named incorrectly.  The intrinsic limits on HDFS files and directories
are a function of namenode memory and our ability to address inodes.  What's being proposed
is an optional setting, so by definition, it's not intrinsic.  Instead, what's being suggested
is a quota (synonym: limit) on the number of files created in a single directory, which translated
to class-speak would be NSPerDirectoryFileQuota.  A case can be made that this would be a
good quota to have, in addition to the existing NSQuota.  Nestled cozily amongst the other
quotas, it would make sense.  But implemented with a different name and separated in the code
from its fellows, it's error prone and confusing.

bq. (I'll avoid the design dispute, and explain why the implementation is the way it is)
The actual implementation is much less important than how this is exposed to the user.  Adding
an extra parameter to updateCount is fine if it is in aid of getting the correct implementation
of this feature.

bq. It is like 'ulimit' and quota on a unix system. They solve different problems. 
Not really.  I was fully expecting another JIRA to implement ulimit-like functionality to
prevent users from opening millions of files at the same, which may be reasonable but is orthogonal
to this.

> Intrinsic limits for HDFS files, directories
> --------------------------------------------
>
>                 Key: HDFS-1751
>                 URL: https://issues.apache.org/jira/browse/HDFS-1751
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1751-2.patch, HDFS-1751-3.patch, HDFS-1751-4.patch, HDFS-1751.patch
>
>
> Enforce a configurable limit on:
>   the length of a path component
>   the number of names in a directory
> The intention is to prevent a too-long name or a too-full directory. This is not about
RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of
limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation
might be to extend the existing quota checker so that it faults the creation of a name that
violates the limits. This strategy of faulting new creation evades the problem of existing
names or directories that violate the limits.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message