hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8137) Sends the EC schema to DataNode as well in EC encoding/recovering command
Date Mon, 04 May 2015 10:28:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526509#comment-14526509
] 

Kai Zheng commented on HDFS-8137:
---------------------------------

Thanks for your update, Uma. The patch looks good and just two minor comments:
* Regarding the following codes, I'm not very sure about the log output. Would it be better
to be:
{code}
blockLog.warn( "Failed to get the ECSchema for the file {} ", filename_of_the_block);
...
blockLog.warn( "No EC Schema found for the file {} ", filename_of_the_block);
{code}
{code}
+            ECSchema ecSchema = null;
+            try {
+              ecSchema = namesystem.getECSchemaForPath(block
+                  .getBlockCollection().getName());
+            } catch (IOException e) {
+              blockLog.warn(
+                  "Failed to get the ECSchema for the blockGroup {} ", block);
+            }
+            if (ecSchema == null) {
+              blockLog.warn("No EC Schema found for the blockGroup {} , "
+                  + "so ignoring the block group for EC", block);
+              // TODO: we may have to revisit later for what we can do better to
+              // handle this case.
+              continue;
+            }
{code}
* In the codes below, might be better to compare ecSchema1 and ecSchema2 with the system default
schema used.
{code}
+    ECSchema ecSchema1 = blkECRecoveryInfo1.getECSchema();
+    ECSchema ecSchema2 = blkECRecoveryInfo2.getECSchema();
+    assertEquals(ecSchema1.getSchemaName(), ecSchema2.getSchemaName());
+    assertEquals(ecSchema1.getNumDataUnits(), ecSchema2.getNumDataUnits());
+    assertEquals(ecSchema1.getNumParityUnits(), ecSchema2.getNumParityUnits());
{code}

> Sends the EC schema to DataNode as well in EC encoding/recovering command
> -------------------------------------------------------------------------
>
>                 Key: HDFS-8137
>                 URL: https://issues.apache.org/jira/browse/HDFS-8137
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Uma Maheswara Rao G
>         Attachments: HDFS-8137-0.patch, HDFS-8137-1.patch
>
>
> Discussed with [~umamaheswararao] and [~vinayrpet], we should also send the EC schema
to DataNode as well contained in the EC encoding/recovering command. The target DataNode will
use it to guide the executing of the task. 
> Another way would be, DataNode would just request schema actively thru a separate RPC
call, and as an optimization consideration, DataNode may cache schemas to avoid repeatedly
asking for the same schema twice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message