hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jinglong.liujl (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1268) Extract blockInvalidateLimit as a seperated configuration
Date Wed, 30 Jun 2010 08:52:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883848#action_12883848

jinglong.liujl commented on HDFS-1268:

Currently, invalidBlocklimit is computed by max (BLOCK_INVALIDATE_CHUNK, 20 * heartbeatInterval),
If I want to modified this parameter, there're two choice.
1. set BLOCK_INVALIDATE_CHUNK  a huge one, and re-compile code
2. increase heartbeat interval, but it can not carry more blocks totally.
what I want is made invalidBlocklimit  can be configed by user.

If our cluster meet this "corner case", (in fact, it's not corner case, we use hbase in this
cluster, this case can be seen very often.), why not config this parameter and restart cluster
to prevent this issue? 

> Extract blockInvalidateLimit as a seperated configuration
> ---------------------------------------------------------
>                 Key: HDFS-1268
>                 URL: https://issues.apache.org/jira/browse/HDFS-1268
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: jinglong.liujl
>         Attachments: patch.diff
>       If there're many file piled up in recentInvalidateSets, only Math.max(blockInvalidateLimit,

> 20*(int)(heartbeatInterval/1000)) invalid blocks can be carried in a heartbeat.(By default,
It's 100). In high write stress, it'll cause process of invalidate blocks removing can not
catch up with  speed of writing. 
>     We extract blockInvalidateLimit  to a sperate config parameter that user can make
the right configure for your cluster. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message