hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4305) Add a configurable limit on number of blocks per file, and min block size
Date Wed, 10 Apr 2013 14:54:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627856#comment-13627856
] 

Kihwal Lee commented on HDFS-4305:
----------------------------------

We had a user who accidentally set the block size to something small in his big job. That
job alone added 10s of millions of blocks when it was still in early stage. The blocks/file
was large but did not grow large enough to cause trouble (0.23 was without HDFS-4304) before
taking an admin action, since the progress was so slow. The blocks were allocated and completed
so frequently that the namenode couldn't keep up, let alone the pressure on the namespace
and heap. 

In cases like this, max blocks per file may not be useful for detecting the issue, but enforcing
min block size will help. 

                
> Add a configurable limit on number of blocks per file, and min block size
> -------------------------------------------------------------------------
>
>                 Key: HDFS-4305
>                 URL: https://issues.apache.org/jira/browse/HDFS-4305
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 1.0.4, 3.0.0, 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Assignee: Andrew Wang
>            Priority: Minor
>         Attachments: hdfs-4305-1.patch
>
>
> We recently had an issue where a user set the block size very very low and managed to
create a single file with hundreds of thousands of blocks. This caused problems with the edit
log since the OP_ADD op was so large (HDFS-4304). I imagine it could also cause efficiency
issues in the NN. To prevent users from making such mistakes, we should:
> - introduce a configurable minimum block size, below which requests are rejected
> - introduce a configurable maximum number of blocks per file, above which requests to
add another block are rejected (with a suitably high default as to not prevent legitimate
large files)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message