hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3335) check for edit log corruption at the end of the log
Date Mon, 14 May 2012 07:58:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Colin Patrick McCabe updated HDFS-3335:

    Attachment: HDFS-3335.011.patch

* increase MAX_OP_SIZE to 100 MB.  Since we are limiting a bunch of strings to 1 MB in size,
and we can have a few of them in an opcode, it only makes sense.

* In the OfflineEditsViewer (oev), we have to initialize the EditLogFileInputStream with startTx
= endTxId = INVALID_TXID.  This is a special value that indicates that we don't know where
the log ends.  Previously, I had been passing -1 as the start and end txid, but the convention
throughout the code is that only INVALID_TXID may be used for this purpose.  This fixes the
unit test failure.
> check for edit log corruption at the end of the log
> ---------------------------------------------------
>                 Key: HDFS-3335
>                 URL: https://issues.apache.org/jira/browse/HDFS-3335
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-3335-b1.001.patch, HDFS-3335-b1.002.patch, HDFS-3335-b1.003.patch,
HDFS-3335-b1.004.patch, HDFS-3335.001.patch, HDFS-3335.002.patch, HDFS-3335.003.patch, HDFS-3335.004.patch,
HDFS-3335.005.patch, HDFS-3335.006.patch, HDFS-3335.007.patch, HDFS-3335.008.patch, HDFS-3335.009.patch,
HDFS-3335.010.patch, HDFS-3335.011.patch
> Even after encountering an OP_INVALID, we should check the end of the edit log to make
sure that it contains no more edits.
> This will catch things like rare race conditions or log corruptions that would otherwise
remain undetected.  They will got from being silent data loss scenarios to being cases that
we can detect and fix.
> Using recovery mode, we can choose to ignore the end of the log if necessary.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message