hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manoj Govindassamy (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10530) BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy
Date Fri, 10 Mar 2017 23:27:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905864#comment-15905864
] 

Manoj Govindassamy edited comment on HDFS-10530 at 3/10/17 11:26 PM:
---------------------------------------------------------------------

[~demongaorui], [~andrew.wang], [~zhz],

The test HDFS-10530.3.patch provided by [~demongaorui] shows the addition of data nodes in
new racks doesn't necessarily rebalance the existing striped blocks for better placement.
The test does the following

1. Start cluster with 6 DNs in 6 Racks
2. Create file with RS-6-3 EC policy. Parity blocks couldn't be created owing to non-availability
of DNs
3. Verify block placement is satisfied as there are only 6 racks available [though the ideal
would be 9]
4. Add 3 more DNs in 3 more Racks.
5. Verify the block placement is still not satisfied for the previously created file
6. Create a new striped file and now verify if the new file's blocks placement policy is satisfied

I modified the test a little more to do the following..

4. Add 3 more DNs in the existing racks
5. Wait for Replication monitor to recover under replicated blocks
6. Parity blocks now gets created on the newly added DNs.
7. Now the file has 9 blocks, in 9 DNs in 6 racks.
8. Verify this block placement is satisfied as there are only 6 Racks
9. Add 3 more DNs in 3 more Racks
10. With NO fix applied, verify this block placement is NOT satisfied as there are sufficient
9 Racks, but blocks are not striped on all.
6. Create a new striped file and verify if the new file's blocks placement policy is satisfied
with striping on all 9 racks.

*Fix Proposal:*

1. The Goal is to find if there are Under-Replicated/Mis-Placed blocks for EC files upon DN
addition on a new Rack. 

2. {{DatanodeManager#addDatanode}} can detect the new topology changes. There is already a
method {{#checkIfClusterIsNowMultiRack}} which was specifically written for Racks growing
from 1 to more. But, I believe we need a more generic check here in the context of EC.

3. {{checkIfClusterIsNowMultiRack}} => {{processMisReplicatedBlocks}} => {{processMisReplicatesAsync}}
is heavy weight as it scans the entire BlockMap. Though rack addition is a rare operation,
doing this whole world scan for every rack addition seems quite time consuming. The ideal
thing would be take some clues and trigger {{processMisReplicatesAsync}} only when needed
so that it detect Mis-placements and trigger proper block placements

4. How about checking for the list of EC policies ever set on any files ? Just like Enabled
EC policy list, we can maintain active EC Policy list (max of sys polices count). This can
serve as a clue for the potential block mis-placements as we can deduce the rack requirements
from the EC policy schema. Once getting the clue, we can trigger  {{processMisReplicatesAsync}}
which can then feed the work to Replication monitor which is running continuously. 

Please share your thoughts on the above proposal. Would love to hear your suggestions on better
alternative approaches. Thanks.



was (Author: manojg):
[~demongaorui], [~andrew.wang], [~zhz],

The test HDFS-10530.3.patch provided by [~demongaorui] shows the addition of data nodes in
new racks doesn't necessarily rebalance the existing striped blocks for better placement.
The test does the following

1. Start cluster with 6 DNs in 6 Racks
2. Create file with RS-6-3 EC policy. Parity blocks couldn't be created owing to non-availability
of DNs
3. Verify block placement is satisfied as there are only 6 racks available [though the ideal
would be 9]
4. Add 3 more DNs in 3 more Racks.
5. Verify the block placement is still not satisfied for the previously created file
6. Create a new striped file and now verify if the new file's blocks placement policy is satisfied

I modified the test a little more to do the following..

4. Add 3 more DNs in the existing racks
5. Wait for Replication monitor to recover under replicated blocks
6. Parity blocks now gets created on the newly added DNs.
7. Now the file has 9 blocks, in 9 DNs in 6 racks.
8. Verify this block placement is satisfied as there are only 6 Racks
9. Add 3 more DNs in 3 more Racks
10. With NO fix applied, verify this block placement is NOT satisfied as there are only 9
Racks, but blocks are not striped on all.
6. Create a new striped file and now verify if the new file's blocks placement policy is satisfied

*Fix Proposal:*

1. The Goal is to find if there are Under-Replicated/Mis-Placed blocks for EC files upon DN
addition on a new Rack. 

2. {{DatanodeManager#addDatanode}} can detect the new topology changes. There is already a
method {{#checkIfClusterIsNowMultiRack}} which was specifically written for Racks growing
from 1 to more. But, I believe we need a more generic check here in the context of EC.

3. {{checkIfClusterIsNowMultiRack}} => {{processMisReplicatedBlocks}} => {{processMisReplicatesAsync}}
is heavy weight as it scans the entire BlockMap. Though rack addition is a rare operation,
doing this whole world scan for every rack addition seems quite time consuming. The ideal
thing would be take some clues and trigger {{processMisReplicatesAsync}} only when needed
so that it detect Mis-placements and trigger proper block placements

4. How about checking for the list of EC policies ever set on any files ? Just like Enabled
EC policy list, we can maintain active EC Policy list (max of sys polices count). This can
serve as a clue for the potential block mis-placements as we can deduce the rack requirements
from the EC policy schema. Once getting the clue, we can trigger  {{processMisReplicatesAsync}}
which can then feed the work to Replication monitor which is running continuously. 

Please share your thoughts on the above proposal. Would love to hear your suggestions on better
alternative approaches. Thanks.


> BlockManager reconstruction work scheduling should correctly adhere to EC block placement
policy
> ------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10530
>                 URL: https://issues.apache.org/jira/browse/HDFS-10530
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Rui Gao
>            Assignee: Rui Gao
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-10530.1.patch, HDFS-10530.2.patch, HDFS-10530.3.patch
>
>
> This issue was found by [~tfukudom].
> Under RS-DEFAULT-6-3-64k EC policy, 
> 1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each) of the
cluster.
> 2. Reconstruction work would be scheduled if the 6th rack is added. 
> 3. While adding the 7th rack or more racks will not trigger reconstruction work. 
> Based on default EC block placement policy defined in “BlockPlacementPolicyRackFaultTolerant.java”,
EC file should be able to be scheduled to distribute to 9 racks if possible.
> In *BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock)* , *numReplicas* of
striped blocks might should be *getRealTotalBlockNum()*, instead of *getRealDataBlockNum()*.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message