hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5517) Lower the default maximum number of blocks per file
Date Fri, 15 Nov 2013 06:37:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823344#comment-13823344

Uma Maheswara Rao G commented on HDFS-5517:

+1 for lowering the value. Thanks Aaron for proposing this change. Recently we also faced
this issue. When file having 2L blocks itself NN is behaving something abnormal.
For every block addition we do update blocks, so this big array will be persisted every time
incrementally I think. This will be overhead really, if allow to write more and more blocks
for same file with the current code But in practice I am not sure what will be the scenario
to configure less block sizes and more blocks for same file. But we saw this in testing cluster.

> Lower the default maximum number of blocks per file
> ---------------------------------------------------
>                 Key: HDFS-5517
>                 URL: https://issues.apache.org/jira/browse/HDFS-5517
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.2.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
> We introduced the maximum number of blocks per file in HDFS-4305, but we set the default
to 1MM. In practice this limit is so high as to never be hit, whereas we know that an individual
file with 10s of thousands of blocks can cause problems. We should lower the default value,
in my opinion to 10k.

This message was sent by Atlassian JIRA

View raw message