hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12570) [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements
Date Thu, 12 Oct 2017 04:04:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198546#comment-16198546
] 

Uma Maheswara Rao G edited comment on HDFS-12570 at 10/12/17 4:03 AM:
----------------------------------------------------------------------

[~rakeshr] Thank you for updating the patch. Patch mostly looks good to me, except the below
comments

* . {code}
public static final String DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA_MAX_STREAMS_KEY
=
+      "dfs.storage.policy.satisfier.share.equal.replication.max-streams";
+  public static final boolean DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA_MAX_STREAMS_DEFAULT
=
+      false;
{code}
I am thinking this config can be more meaningful. How about something like, "dfs.storage.policy.satisfier.low.max-streams.preference”.
This can be true by default. Any other better names are most welcomed.
* finshedBlksIter — finishedBlksIter
* We are removing the state: status = BlocksMovingAnalysisStatus.FEW_BLOCKS_TARGETS_PAIRED;
how are we covering the case when we can’t find any targets at all even though there are
movement needed blocks?
What is the status for this?
* . {code}
 // 'BlockCollectionId' is used as the tracking ID. All the blocks under
+        // this
+        // blockCollectionID will be added to this datanode.
+        // TODO: assign to each target node
{code}
Please remove this comment or update.
* . {code}
+        ((DatanodeDescriptor) blkMovingInfo.getTarget())
+            .addBlocksToMoveStorage(blkMovingInfo);
{code}
Could we also increment scheduled block count for this node?
* BlockStorageMovementCommand: We removed tracked, but class java doc still representing old
behavior.
* Documentation needs update.


was (Author: umamaheswararao):
[~rakeshr] Thank you for updating the patch. Patch mostly looks good to me, except the below
comments

* . {code}
public static final String DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA_MAX_STREAMS_KEY
=
+      "dfs.storage.policy.satisfier.share.equal.replication.max-streams";
+  public static final boolean DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA_MAX_STREAMS_DEFAULT
=
+      false;
{code}
I am thinking this config can be more meaningful. How about something like, "dfs.storage.policy.satisfier.low.max-streams.preference”.
This can be true by default. Any other better names are most welcomed.
* finshedBlksIter — finihedBlksIter
* We are removing the state: status = BlocksMovingAnalysisStatus.FEW_BLOCKS_TARGETS_PAIRED;
how are we covering the case when we can’t find any targets at all even though there are
movement needed blocks?
What is the status for this?
* . {code}
 // 'BlockCollectionId' is used as the tracking ID. All the blocks under
+        // this
+        // blockCollectionID will be added to this datanode.
+        // TODO: assign to each target node
{code}
Please remove this comment or update.
* . {code}
+        ((DatanodeDescriptor) blkMovingInfo.getTarget())
+            .addBlocksToMoveStorage(blkMovingInfo);
{code}
Could we also increment scheduled block count for this node?
* BlockStorageMovementCommand: We removed tracked, but class java doc still representing old
behavior.

> [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-12570
>                 URL: https://issues.apache.org/jira/browse/HDFS-12570
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-12570-HDFS-10285-00.patch, HDFS-12570-HDFS-10285-01.patch,
HDFS-12570-HDFS-10285-02.patch
>
>
> This task is to refactor the C-DN block storage movements. Basically, the idea is to
move the scheduling and tracking logic to Namenode rather than at the special C-DN. Please
refer the discussion with [~andrew.wang] to understand the [background and the necessity of
refactoring|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16141060&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16141060].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message