hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks
Date Sat, 18 Apr 2015 00:51:59 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jing Zhao updated HDFS-8145:
----------------------------
    Attachment: HDFS-8145.002.patch

Thanks for the review, Zhe! Update the patch to address your comments:
# I just removed isInECZone method from ECZoneManager since it is only called once there.
Also in this way in FSDirectory#isInECZone we do not need to acquire the read lock again.
# Actually for editlog transaction we do not need to persist these two numbers even if later
we no longer use the hard-coded numbers. For both {{addBlock}} and {{updateBlocks}}, the schema
information can be retrieved from the corresponding EC zone. We need to write schema information
for INode only if we decide to drop the ECZone design, and that change will have a much bigger
scope and a lot of pending issues in my opinion.

> Fix the editlog corruption exposed by failed TestAddStripedBlocks
> -----------------------------------------------------------------
>
>                 Key: HDFS-8145
>                 URL: https://issues.apache.org/jira/browse/HDFS-8145
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-8145.000.patch, HDFS-8145.001.patch, HDFS-8145.002.patch
>
>
> {{TestAddStripedBlocks}} failed with some editlog corruption. Did some debugging, I can
see at least two issues:
> # DFSStripedOutputStream tries to send out an empty packet to close the block even if
writing 0 bytes
> # Because of the above, NN tries to close the file. This exposes another bug in {{BlockInfoStriped}},
which writes its data/parity block numbers into the close editlog but do not read them while
loading.
> This jira aims to only fix #2 and HDFS-8166 is used for fixing #1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message