hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11068) [SPS]: Provide unique trackID to track the block movement sends to coordinator
Date Thu, 10 Nov 2016 03:12:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15652862#comment-15652862

Uma Maheswara Rao G commented on HDFS-11068:


+  public Map<Long, List<BlockMovingInfo>> getBlocksToMoveStorages() {
+    Map<Long, List<BlockMovingInfo>> trackIdVsBlocks = new LinkedHashMap<>();
+    synchronized (storageMovementBlocks) {
+      if (storageMovementBlocks.isEmpty()) {
+        return trackIdVsBlocks;
+      }
+      trackIdVsBlocks.putAll(storageMovementBlocks);
+      storageMovementBlocks.keySet().removeAll(trackIdVsBlocks.keySet());
+    }
+    return trackIdVsBlocks;
Here what if one trackId/blockcollection contains many blocks to move? So, how about just
keep once trackID per heartbeat? (Later we may need to sub decid them into small batches with
in tracked itself if blocks are many(ex: a file contains many blocks)) 
+      // TODO: Temporarily using the results from StoragePolicySatisfier
+      // class. This has to be revisited as part of HDFS-11029.
HDFS-11029 almost ready. we can incorporate required changes and remove this TODO. Thanks
for adding TODO.

Other than this comments, patch looks great. Thanks

> [SPS]: Provide unique trackID to track the block movement sends to coordinator
> ------------------------------------------------------------------------------
>                 Key: HDFS-11068
>                 URL: https://issues.apache.org/jira/browse/HDFS-11068
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-11068-HDFS-10285-01.patch, HDFS-11068-HDFS-10285.patch
> Presently DatanodeManager uses constant  value -1 as [trackID|https://github.com/apache/hadoop/blob/HDFS-10285/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java#L1607],
which is a temporary value. As per discussion with [~umamaheswararao], one proposal is to
use {{BlockCollectionId/InodeFileId}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message