Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 12 Oct 2017 04:04:00 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13106204.1506750793000.36003.1507781040712@Atlassian.JIRA>
In-Reply-To: <JIRA.13106204.1506750793000@Atlassian.JIRA>
References: <JIRA.13106204.1506750793000@Atlassian.JIRA> <JIRA.13106204.1506750793077@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (HDFS-12570) [SPS]: Refactor Co-ordinator
 datanode logic to track the block storage movements
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 12 Oct 2017 04:04:13 -0000


    [ https://issues.apache.org/jira/browse/HDFS-12570?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1619=
8546#comment-16198546 ]=20

Uma Maheswara Rao G edited comment on HDFS-12570 at 10/12/17 4:03 AM:
----------------------------------------------------------------------

[~rakeshr] Thank you for updating the patch. Patch mostly looks good to me,=
 except the below comments

* . {code}
public static final String DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA=
_MAX_STREAMS_KEY =3D
+      "dfs.storage.policy.satisfier.share.equal.replication.max-streams";
+  public static final boolean DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REP=
LICA_MAX_STREAMS_DEFAULT =3D
+      false;
{code}
I am thinking this config can be more meaningful. How about something like,=
 "dfs.storage.policy.satisfier.low.max-streams.preference=E2=80=9D. This ca=
n be true by default. Any other better names are most welcomed.
* finshedBlksIter =E2=80=94 finishedBlksIter
* We are removing the state: status =3D BlocksMovingAnalysisStatus.FEW_BLOC=
KS_TARGETS_PAIRED;
how are we covering the case when we can=E2=80=99t find any targets at all =
even though there are movement needed blocks?
What is the status for this?
* . {code}
 // 'BlockCollectionId' is used as the tracking ID. All the blocks under
+        // this
+        // blockCollectionID will be added to this datanode.
+        // TODO: assign to each target node
{code}
Please remove this comment or update.
* . {code}
+        ((DatanodeDescriptor) blkMovingInfo.getTarget())
+            .addBlocksToMoveStorage(blkMovingInfo);
{code}
Could we also increment scheduled block count for this node?
* BlockStorageMovementCommand: We removed tracked, but class java doc still=
 representing old behavior.
* Documentation needs update.


was (Author: umamaheswararao):
[~rakeshr] Thank you for updating the patch. Patch mostly looks good to me,=
 except the below comments

* . {code}
public static final String DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REPLICA=
_MAX_STREAMS_KEY =3D
+      "dfs.storage.policy.satisfier.share.equal.replication.max-streams";
+  public static final boolean DFS_STORAGE_POLICY_SATISFIER_SHARE_EQUAL_REP=
LICA_MAX_STREAMS_DEFAULT =3D
+      false;
{code}
I am thinking this config can be more meaningful. How about something like,=
 "dfs.storage.policy.satisfier.low.max-streams.preference=E2=80=9D. This ca=
n be true by default. Any other better names are most welcomed.
* finshedBlksIter =E2=80=94 finihedBlksIter
* We are removing the state: status =3D BlocksMovingAnalysisStatus.FEW_BLOC=
KS_TARGETS_PAIRED;
how are we covering the case when we can=E2=80=99t find any targets at all =
even though there are movement needed blocks?
What is the status for this?
* . {code}
 // 'BlockCollectionId' is used as the tracking ID. All the blocks under
+        // this
+        // blockCollectionID will be added to this datanode.
+        // TODO: assign to each target node
{code}
Please remove this comment or update.
* . {code}
+        ((DatanodeDescriptor) blkMovingInfo.getTarget())
+            .addBlocksToMoveStorage(blkMovingInfo);
{code}
Could we also increment scheduled block count for this node?
* BlockStorageMovementCommand: We removed tracked, but class java doc still=
 representing old behavior.

> [SPS]: Refactor Co-ordinator datanode logic to track the block storage mo=
vements
> -------------------------------------------------------------------------=
-------
>
>                 Key: HDFS-12570
>                 URL: https://issues.apache.org/jira/browse/HDFS-12570
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-12570-HDFS-10285-00.patch, HDFS-12570-HDFS-1028=
5-01.patch, HDFS-12570-HDFS-10285-02.patch
>
>
> This task is to refactor the C-DN block storage movements. Basically, the=
 idea is to move the scheduling and tracking logic to Namenode rather than =
at the special C-DN. Please refer the discussion with [~andrew.wang] to und=
erstand the [background and the necessity of refactoring|https://issues.apa=
che.org/jira/browse/HDFS-10285?focusedCommentId=3D16141060&page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16141060].


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org