hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jinglong.liujl (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1268) Extract blockInvalidateLimit as a seperated configuration
Date Mon, 28 Jun 2010 15:08:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883176#action_12883176
] 

jinglong.liujl commented on HDFS-1268:
--------------------------------------

In my case, if I want to delete 600 blocks,  I have to wait 6 heartbeats periods. During this
period,  disk maybe reach its capacity. Then, too slow block fetching will cause write failure.

In general case, default value (100 can work well), but in this extremely case, default
value is not enough. Currently, this parameter can be computed by heartbeatInterval, but in
the case before, "slower heartbeat + per heartbeat carry more blocks " can not carry more
blocks in the same period.
Why not make this parameter can be configured?





> Extract blockInvalidateLimit as a seperated configuration
> ---------------------------------------------------------
>
>                 Key: HDFS-1268
>                 URL: https://issues.apache.org/jira/browse/HDFS-1268
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: jinglong.liujl
>         Attachments: patch.diff
>
>
>       If there're many file piled up in recentInvalidateSets, only Math.max(blockInvalidateLimit,

> 20*(int)(heartbeatInterval/1000)) invalid blocks can be carried in a heartbeat.(By default,
It's 100). In high write stress, it'll cause process of invalidate blocks removing can not
catch up with  speed of writing. 
>     We extract blockInvalidateLimit  to a sperate config parameter that user can make
the right configure for your cluster. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message