hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surendra Singh Lilhore (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11726) [SPS] : StoragePolicySatisfier should not select same storage type as source and destination in same datanode.
Date Sun, 28 May 2017 13:30:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16027821#comment-16027821
] 

Surendra Singh Lilhore commented on HDFS-11726:
-----------------------------------------------

I got the root cause for this.

SPS first give the preference to the local DN. If expected storage type is not exist in source
dn then it will select remote DN. We missed one check in StoragePolicySatisfier#findSourceAndTargetToMove().
If replica with expected storage type already exist in source DN then we should skip that
DN.
For example:

Expected storage is : ARCHIVE
Source DN with storage : DN1(ARCHIVE), DN2(DISK).

We should not select DN1 for block movement because here already replica is exist in ARCHIVE.



> [SPS] : StoragePolicySatisfier should not select same storage type as source and destination
in same datanode.
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-11726
>                 URL: https://issues.apache.org/jira/browse/HDFS-11726
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: HDFS-10285
>            Reporter: Surendra Singh Lilhore
>            Assignee: Surendra Singh Lilhore
>
> {code}
> 2017-04-30 16:12:28,569 [BlockMoverTask-0] INFO  datanode.StoragePolicySatisfyWorker
(Worker.java:moveBlock(248)) - Start moving block:blk_1073741826_1002 from src:127.0.0.1:41699
to destin:127.0.0.1:41699 to satisfy storageType, sourceStoragetype:ARCHIVE and destinStoragetype:ARCHIVE
> {code}
> {code}
> 2017-04-30 16:12:28,571 [DataXceiver for client /127.0.0.1:36428 [Replacing block BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002
from 6c7aa66e-a778-43d5-89f6-053d5f6b35bc]] INFO  datanode.DataNode (DataXceiver.java:replaceBlock(1202))
- opReplaceBlock BP-1409501412-127.0.1.1-1493548923222:blk_1073741826_1002 received exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica,
blk_1073741826_1002, FINALIZED
>   getNumBytes()     = 1024
>   getBytesOnDisk()  = 1024
>   getVisibleLength()= 1024
>   getVolume()       = /home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7
>   getBlockURI()     = file:/home/sachin/software/hadoop/HDFS-10285/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7/current/BP-1409501412-127.0.1.1-1493548923222/current/finalized/subdir0/subdir0/blk_1073741826
already exists on storage ARCHIVE
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message