hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-9671) DiskBalancer : SubmitPlan implementation
Date Thu, 18 Feb 2016 01:16:18 GMT

     [ https://issues.apache.org/jira/browse/HDFS-9671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Anu Engineer updated HDFS-9671:
    Attachment: HDFS-9671-HDFS-1312.006.patch

bq. DiskBalancer#shutdown should wait for the executor to terminate following the convention
in DataNode#shutdown. e.g. DirectoryScanner#shutdown.

Thanks for catching this. I have addressed this isssue by adding a proper 
shutdown function.

bq. Related to the volume references mentioned by Lei (Eddy) Xu, it looks safer to use storageuuids
in DiskBalancer and never cache volume references. I assume you'd call into the dataset to
perform the actual move at some point. We could defer the storageuuid-->volume resolution
to the dataset until then.

We do exactly that in createPlan(NodePlan plan)

bq. blockMover.setExitFlag() looks unnecessary. scheduler.shutdownNow() will cancel it as
long as the BlockMover implementation does not swallow InterruptedException. I could have
failed to think of some case here.

The BlockMover does catch the InterruptedException, hence we are doing the setExitFlag.

bq. A future improvement - can we reclaim the thread once the executor has finished executing
a plan? This reminds me to fix an extra thread I recently added to the NN. 

will chat with you off-line on how to get this done.

bq. Spurious edits to the license header in DiskBalancerException.java?

bq. verifyTimeStamp - 24 hours limit should be a constant and probably documented somewhere.
Also we can use TimeUnit.DAYS.toMillis. Ok with handling in a subsequent Jira.


> DiskBalancer : SubmitPlan implementation 
> -----------------------------------------
>                 Key: HDFS-9671
>                 URL: https://issues.apache.org/jira/browse/HDFS-9671
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>    Affects Versions: HDFS-1312
>            Reporter: Anu Engineer
>            Assignee: Anu Engineer
>         Attachments: HDFS-9671-HDFS-1312.001.patch, HDFS-9671-HDFS-1312.002.patch, HDFS-9671-HDFS-1312.003.patch,
HDFS-9671-HDFS-1312.004.patch, HDFS-9671-HDFS-1312.005.patch, HDFS-9671-HDFS-1312.006.patch
> Datanode side code for submit plan for diskbalancer.

This message was sent by Atlassian JIRA

View raw message