hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1195) Offer rate limits for replicating data
Date Wed, 09 Jun 2010 16:43:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877128#action_12877128
] 

Eli Collins commented on HDFS-1195:
-----------------------------------

You can throttle replication indirectly by raising dfs.namenode.replication.interval (the
frequency in seconds for when the namenode should check for new blocks to replicate). Each
interval, every 3s by default, the namenode schedules 2 x # DNs blocks to replicate. Being
able to throttle it directly via a rate would be nice though.

> Offer rate limits for replicating data 
> ---------------------------------------
>
>                 Key: HDFS-1195
>                 URL: https://issues.apache.org/jira/browse/HDFS-1195
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.20.2
>         Environment: Linux, Hadoop 0.20.1 CDH
>            Reporter: Kevin Weil
>
> If a rack of Hadoop nodes goes down, there is a lot of data to re-replicate.  It would
be great to have a configuration option to rate-limit the amount of bandwidth used for re-replication
so as not to saturate network backlinks.  There is a similar option for rate limiting the
speed at which a DFS rebalance takes place: dfs.balance.bandwidthPerSec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message