hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8120) Erasure coding: created util class to analyze striped block groups
Date Wed, 15 Apr 2015 01:21:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495520#comment-14495520
] 

Jing Zhao commented on HDFS-8120:
---------------------------------

Thanks Zhe. The latest patch looks pretty good to me. Some nits:
# Let's write the following code using if-else:
{code}
+        boolean wrongSize = storedBlock.getNumBytes() != reported.getNumBytes();
+        if (storedBlock.isStriped()) {
{code}
i.e., 
{code}
boolean wrongSize;
if (storedBlock.isStriped()) {
  // ....
} else {
  // ....
}
{code}
# In {{DFSTestUtil#createStripedFile}}, instead of using null to indicate
  no need to create directory and EC zone, it may be better to use an additional
  parameter {{toMkdir}}.
# Nit: need to remove 2 spaces before "@Test".
{code}
-  //  @Test
+    @Test
   public void TestFileMoreThanABlockGroup2() throws IOException {
{code}

Besides, {{TestAddStripedBlocks}} failed with some editlog corruption. Did some debugging,
I can see at least two issues:
# DFSStripedOutputStream tries to send out an empty packet to close the block even if writing
0 bytes
# Because of the above bug, NN tries to close the file. This exposes another bug in {{BlockInfoStriped}},
which writes its data/parity block numbers into the close editlog but do not read them while
loading.

I will create another jira to fix this.

> Erasure coding: created util class to analyze striped block groups
> ------------------------------------------------------------------
>
>                 Key: HDFS-8120
>                 URL: https://issues.apache.org/jira/browse/HDFS-8120
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8120.000.patch, HDFS-8120.001.patch, HDFS-8120.002.patch, HDFS-8120.003.patch,
HDFS-8120.004.patch
>
>
> The patch adds logic of calculating size of individual blocks in a striped block group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message