hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1448) Setting the replication factor of a file too high causes namenode cpu overload
Date Thu, 31 May 2007 19:08:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500452

Doug Cutting commented on HADOOP-1448:

Other ideas:

1. Limit the maximum replication to something reasonably small.

2. When a file is highly-replicated, sample the replicas rather than sorting and returning
the full list.  For example, one could first try to return only the first-N replicas on the
same rack as the client.  If that fails, then the off-rack replicas might be randomly sampled.

Do we think it is reasonable to set replication to numbers proportional to the cluster size?
 If so, do we think it is reasonable for namenode requests to take time proportional to the
cluster size?  If not, then, in these cases, we should try to never return or process the
full list, but rather only a fixed-size subset of it.

> Setting the replication factor of a file too high causes namenode cpu overload
> ------------------------------------------------------------------------------
>                 Key: HADOOP-1448
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1448
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
> The replication factor of a file in set to 300 (on a 800 node cluster). Then all mappers
try to open this file. For every open call that the namenode receives from each of these 800
clients, it sorts all the replicas of the block(s) based on the distance from the client.
This causes CPU usage overload on the namenode.
> One proposal is to make the namenode return a non-sorted list of datanodes to the client.
Information about each replica also contains the rack on which that replica resides. The client
can look at the replicas to determine if there is a copy on the local node. If not, then it
can find out if there is a replica on the local rack. If not then it can choose a replica
at random.
> This proposal is scalable because the sorting and selection of replicas is done by the
client rather than the Namenode.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message