hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3900) QJM: avoid validating log segments on log rolls
Date Fri, 07 Sep 2012 21:32:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451001#comment-13451001

Todd Lipcon commented on HDFS-3900:

bq. One question: note that the refactor into the abortCurSegment() method also causes the
code in newEpoch to now call "curSegment.abort()" instead of "curSegment.close()". Is there
any semantic change there that we should be concerned about?

I noticed this as well, but I determined it doesn't really matter. The only difference is
that abort() leaves the preallocated padding at the end of the file, whereas close() truncates.
Really it might make sense to change all of the abort() calls to close() but I didn't want
to change that in this JIRA. But semantically, we never use the byte-wise length, just the
valid-txn-length, so it doesn't really matter.
> QJM: avoid validating log segments on log rolls
> -----------------------------------------------
>                 Key: HDFS-3900
>                 URL: https://issues.apache.org/jira/browse/HDFS-3900
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: QuorumJournalManager (HDFS-3077)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-3900.txt
> Currently, we are paranoid and validate every log segment when it is finalized. For the
a log segment that has been written entirely by one writer, with no recovery in between, this
is overly paranoid (we don't do this for local journals). It also causes log rolls to be slow
and take time linear in the size of the segment. Instead, we should optimize this path to
simply trust that the segment is correct so long as the txids match up as expected.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message