hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9543) DiskBalancer : Add Data mover
Date Sun, 24 Apr 2016 20:44:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255715#comment-15255715

Anu Engineer commented on HDFS-9543:

[~arpitagarwal] Thanks for the code review comments. I will post a revised patch soon.

bq.  I expect this situation will be seen only if the storage type of a disk has changed since
the planner generated the plan?

We allow disk balancer plans to be edited if needed. This prevents users from making a mistake.
We can avoid it but produces no harm.  Please let me know if you need this removed.

bq. That's a heavyweight lock to hold for the move. I don't think we should synchronize on
the Dataset. 

Completely agree. I did this to play fair with DirectoryScanner. This lock is being held by
directoryScanner to prevent changes to blockMap. I wanted to make sure that diskBalancer and
DirectoryScanner play nicely. Please look at {{DirectoryScanner#scan line 586}}. Please do
let me know if this is not needed in DiskBalancer.

bq.  think computeDelay should substract the time taken for the move else the computed delay
will be too conservative. What do you think?
good catch. Thank you, I will fix this in next update.

bq.  Precondition.checkState is redundant, subsequent lines will NPE if the either object
is null.

Agree and Precondition is going to throw somethign similar. But it does make the invariant
clear to the next person who is reading code instead of wondering if I forgot to check a condition.

bq. I did not get this TODO. What does it mean to move blocks across block pools

I will make this comment clearer in the next update. Balancer supports a flag called -blockpools
-- HDFS-8890 seems to have added this. Just leaving a note that we don't do this yet.

> DiskBalancer : Add Data mover 
> ------------------------------
>                 Key: HDFS-9543
>                 URL: https://issues.apache.org/jira/browse/HDFS-9543
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Anu Engineer
>            Assignee: Anu Engineer
>         Attachments: HDFS-9543-HDFS-1312.001.patch
> This patch adds the actual mover logic to the datanode.

This message was sent by Atlassian JIRA

View raw message