hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3335) check for edit log corruption at the end of the log
Date Wed, 02 May 2012 01:40:48 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Colin Patrick McCabe updated HDFS-3335:
---------------------------------------

    Attachment: HDFS-3335.001.patch

Add patch for trunk.

* Always check for corruption at the end of the log, unless we are reading the last megabyte
of an unfinalized edit log.

* Create constant for PREALLOCATION_LENGTH.  Use it everywhere we were previously copying-and-pasting
the integer.

* FSEditLogLoader: log any exceptions that we get when reading the edit log.

* FSEditLogOp: add verifyEndOfLog

* FSEditLogOp: restrict serialized FSEditLogOp operations to a megabyte in size.
Previously, I was trying to avoid having a hard limit on edit log operation size by doing
{code}
in.mark(in.available())
{code}

However, this is not valid.  in.available() does NOT reflect how many bytes remain in the
stream-- it just reflects the number that can be read without blocking, a number which may
or may not be related to the amount left.
                
> check for edit log corruption at the end of the log
> ---------------------------------------------------
>
>                 Key: HDFS-3335
>                 URL: https://issues.apache.org/jira/browse/HDFS-3335
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-3335-b1.001.patch, HDFS-3335-b1.002.patch, HDFS-3335-b1.003.patch,
HDFS-3335.001.patch
>
>
> Even after encountering an OP_INVALID, we should check the end of the edit log to make
sure that it contains no more edits.
> This will catch things like rare race conditions or log corruptions that would otherwise
remain undetected.  They will got from being silent data loss scenarios to being cases that
we can detect and fix.
> Using recovery mode, we can choose to ignore the end of the log if necessary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message