hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1562) Add rack policy tests
Date Mon, 03 Jan 2011 19:14:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976887#action_12976887

Todd Lipcon commented on HDFS-1562:

Few notes:
- in TestBlocksWithNotEnoughRacks.waitForReplication, I think you need to take the FSN lock
before calling things like blockManager.countNodes, neededReplications.size, etc, since those
functions aren't inherently threadsafe.
- While you're at it, maybe change the while loop to do { ... } while () so you don't need
to duplicate the assignments of curRacks, curReplicas, curNeededReplicas?
- waitForDecomission will wait forever - it should have a timeout after some number of seconds
like waitReplication does.
- testUnderReplicatedUsesNewRacks: would be good to add another test like this with replication
1, increase to 2 - then it would be a substantially different test than the one above, because
it doesn't start out in neededReplication, right?
- testCorruptBlockRereplicatedAcrossRacks: any way to verify here that it ended up with a
clean copy of the block? ie that it's not just claiming "oh yea, I'm on 2 racks" even though
rack2 is a corrupt copy. Perhaps this test could be done by actually corrupting the block
on disk (there's some code for this in one of the various test utils), then triggering the
DN's block scanner. Then wait for it to fix itself.
- "block that is across racks is not removed when deleting replicas" - s/block/replica/
- "Test that when the execss replicas of" - typo "excess"

> Add rack policy tests
> ---------------------
>                 Key: HDFS-1562
>                 URL: https://issues.apache.org/jira/browse/HDFS-1562
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: name-node, test
>    Affects Versions: 0.23.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hdfs-1562-1.patch
> The existing replication tests (TestBlocksWithNotEnoughRacks, TestPendingReplication,
TestOverReplicatedBlocks, TestReplicationPolicy, TestUnderReplicatedBlocks, and TestReplication)
are missing tests for rack policy violations.  This jira adds the following tests which I
created when generating a new patch for HDFS-15.
> * Test that blocks that have a sufficient number of total replicas, but are not replicated
cross rack, get replicated cross rack when a rack becomes available.
> * Test that new blocks for an underreplicated file will get replicated cross rack. 
> * Mark a block as corrupt, test that when it is re-replicated that it is still replicated
across racks.
> * Reduce the replication factor of a file, making sure that the only block that is across
racks is not removed when deleting replicas.
> * Test that when a block is replicated because a replica is lost due to host failure
the the rack policy is preserved.
> * Test that when the execss replicas of a block are reduced due to a node re-joining
the cluster the rack policy is not violated.
> * Test that rack policy is still respected when blocks are replicated due to node decommissioning.
> * Test that rack policy is still respected when blocks are replicated due to node decommissioning,
even when the blocks are over-replicated.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message