hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2018) Move all journal stream management code into one place
Date Mon, 20 Jun 2011 22:53:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052262#comment-13052262
] 

Todd Lipcon commented on HDFS-2018:
-----------------------------------

Some comments on this patch:
- general idea seems right...
- I think some of it overlaps with HDFS-2085 and HDFS-2074, which are awaiting review. Can
you take a look at those?
- for getEditLogManifest I think you need to support the case that different journal managers
will have different sets of logs, but we need to be able to transfer all of them. ie imagine
the case with two edits directories where one fails, comes back, then the other fails. In
that case you need to interleave copying txns from both of them when transferring edits to
the 2NN.
- I just opened HDFS-2088 and about to put a patch up there in a few minutes. That deals with
the archiving logic and makes some similar changes (eg refactoring some stuff out of FSImageTransactionalStorageInspector
into FileJournalManager)

Let me see if I can merge some of your work into my branch -- sorry that I'm a few patches
ahead of what's been committed.

> Move all journal stream management code into one place
> ------------------------------------------------------
>
>                 Key: HDFS-2018
>                 URL: https://issues.apache.org/jira/browse/HDFS-2018
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: Edit log branch (HDFS-1073)
>
>         Attachments: HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff
>
>
> Currently in the HDFS-1073 branch, the code for creating output streams is in FileJournalManager
and the code for input streams is in the inspectors. This change does a number of things.
>   - Input and Output streams are now created by the JournalManager.
>   - FSImageStorageInspectors now deals with URIs when referring to edit logs
>   - Recovery of inprogress logs is performed by counting the number of transactions instead
of looking at the length of the file.
> The patch for this applies on top of the HDFS-1073 branch + HDFS-2003 patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message