hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1751) Intrinsic limits for HDFS files, directories
Date Tue, 15 Mar 2011 23:55:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007289#comment-13007289

Jakob Homan commented on HDFS-1751:

My main concern with this patch centers around the maximum number of files in a directory
ability quota that is being added.  This is a type of quota (a subset of the namespace quota)
but is being treated completely differently from that quota.  For instance, the maximum number
of files is specified via a config param: dfs.namenode.fs-limits.max-directory-contents, whereas
none of the other quotas are settable via config.  The config name itself is problematic:
dfs.namenode.fs-limits.max-directory-contents.  Contents is ambiguous: bytes, namespace, etc?
It should be something along per-directory-nsquota.  

Moreover, this quota is not settable via the standard quotable commands, nor is it documented
at all in the forrest documents.

I'm actually a fan of the specific types of implementations.  We'll get a big win from strongly
typed exceptions instead of string-ly typed ones.

If we're going to introduce a new quota, it should follow the patterns of the previous ones.

> Intrinsic limits for HDFS files, directories
> --------------------------------------------
>                 Key: HDFS-1751
>                 URL: https://issues.apache.org/jira/browse/HDFS-1751
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>         Attachments: HDFS-1751-2.patch, HDFS-1751.patch
> Enforce a configurable limit on:
>   the length of a path component
>   the number of names in a directory
> The intention is to prevent a too-long name or a too-full directory. This is not about
RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of
limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation
might be to extend the existing quota checker so that it faults the creation of a name that
violates the limits. This strategy of faulting new creation evades the problem of existing
names or directories that violate the limits.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message