hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7891) A block placement policy with best fault tolerance
Date Wed, 25 Mar 2015 18:47:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380482#comment-14380482

Zhe Zhang commented on HDFS-7891:

Thanks Nicholas and Walter for the discussion!

bq. I think when we use stripe layout, even when stripe layout is not used. All 6+3 replica(it's
better called blocks belonging to same group) comes from client. As long as the replica is
place on the remote rack, the traffic is the same.
Under the striping layout, in _initial file creation_, all raw or parity data comes directly
from client. So inter-rack traffic won't be affected by placement policy (unless we consider
the factor that client is likely co-located with 1st DN). 

Two other scenarios do incur inter-DN traffic:
# Background block reconstruction. Here a new DN is chosen as reconstruction destination and
will pull data from other DNs in the group
# EC+contiguous layout. Here all blocks are initially triplicated. Then for each formed group,
a new DN is selected as the encoder, which pulls data from raw data blocks

> A block placement policy with best fault tolerance
> --------------------------------------------------
>                 Key: HDFS-7891
>                 URL: https://issues.apache.org/jira/browse/HDFS-7891
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>         Attachments: HDFS-7891.002.patch, HDFS-7891.003.patch, HDFS-7891.004.patch, HDFS-7891.patch,
PlacementPolicyBenchmark.txt, testresult.txt
> a block placement policy tries its best to place replicas to most racks.

This message was sent by Atlassian JIRA

View raw message