hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8178) QJM doesn't purge empty and corrupt inprogress edits files
Date Thu, 30 Apr 2015 00:37:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520617#comment-14520617

Aaron T. Myers commented on HDFS-8178:

Hi Zhe, thanks a lot for posting this patch. It looks pretty good to me. I have one comment,
and one question for you:

Comment: I'm a tad concerned that the current patch be be over aggressive in terms of purging
edit log segments, if somehow we ended up with two in-progress segments, both of which were
required to form the complete history of edit log transactions. Unless we're somehow guaranteed
that that's not a state we could end up in, I think we should do something to guarantee that
the in-progress edits file(s) that we're considering purging definitely overlap with finalized
edit log segments so that they don't contain any edits that we can't afford to lose. Or, perhaps
we should move them aside with a different name until such time as we can be sure that we
don't need their transactions anymore, i.e. their transaction is less than minTxIdToKeep,
and so we can definitely safely discard them.

Question: In the patch for HDFS-5919, we introduced a new regex for identifying stale in-progress
files. I'm not familiar with why that was necessary in that patch, but can you please comment
on why it's not necessary in this case? Naively, I'd expect either both or neither the FJM
and JN to require accounting for that.

> QJM doesn't purge empty and corrupt inprogress edits files
> ----------------------------------------------------------
>                 Key: HDFS-8178
>                 URL: https://issues.apache.org/jira/browse/HDFS-8178
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: qjm
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8178.000.patch
> HDFS-5919 fixes the issue for {{FileJournalManager}}. A similar fix is needed for QJM.

This message was sent by Atlassian JIRA

View raw message