hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8147) Mover should not select the DN storage as target where already same replica exists.
Date Sat, 18 Apr 2015 20:16:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501571#comment-14501571
] 

Tsz Wo Nicholas Sze commented on HDFS-8147:
-------------------------------------------

> ... Mover will not use this API.

No, Mover does use it.  It calls source.addPendingMove(db, target) to create the PendingMove.
source.addPendingMove(db, target) 
=> target.addPendingMove(block, new PendingMove(this, target)
=> pm.markMovedIfGoodBlock(block, getStorageType())
=> isGoodBlockCandidate(source, target, targetStorageType, block)

If your patch is working, could you add a test to verify it?


> Mover should not select the DN storage as target where already same replica exists.
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-8147
>                 URL: https://issues.apache.org/jira/browse/HDFS-8147
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover
>    Affects Versions: 2.6.0
>            Reporter: surendra singh lilhore
>            Assignee: surendra singh lilhore
>         Attachments: HDFS-8147.patch
>
>
> *Scenario:*
> 1. Three DN cluster.  For DNs storage type is like this.
>         DN1 : DISK,ARCHIVE
>         DN2 : DISK
>         DN3 : DISK,ARCHIVE (All DNs are in same rack)
> 2. One file with two replicas (In DN1 and DN2)
> 3. Set file storage policy COLD
> 4. Now execute Mover.
> *Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
> *Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to D1:ARCHIVE, but in
next iteration {{chooseTarget()}} for same rake is selecting again DN1:ARCHIVE for target
where already same block exists.
> {{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the node as target
where already same replica exists.
> *Logs*
> {code}
> 15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990
from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error,
status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028
received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message