hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-12270) Allow more spreading of replicas during block placement
Date Mon, 07 Aug 2017 18:30:00 GMT
Kihwal Lee created HDFS-12270:

             Summary: Allow more spreading of replicas during block placement
                 Key: HDFS-12270
                 URL: https://issues.apache.org/jira/browse/HDFS-12270
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: block placement
            Reporter: Kihwal Lee

The default block placement places the first replica locally if possible, then on a node in
a remote rack, and finally another node in the remote rack. If more than 3 replicas are requested,
the rest are spread across available racks.  This strategy was chosen to minimize the inter-rack
traffic and be able to tolerate a rack-level failure such as switch outages.

This can tolerate a single rack failure, but if there also is a node outage (double failure),
having missing blocks is highly likely. Although network bandwidth is still limited resource,
it is less so than in the past. Some users might want increased data availability at the price
of increased inter-rack traffic.  

This can be achieved by using the upgrade domain feature, but a simple tweak in the default
policy can enable this, in case one does not want to go with the upgrade domain.

I propose introducing a new config to control this.
Rack placement level 0: default. Current behavior.
Rack placement level 1: Use minimum 3 racks, if available. Allow existing blocks to remain
as is.
Rack placement level 2: Use minimum 3 racks, if available. Apply this policy to all replication
verification. (e.g. replication queue initialization)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message