hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2018) 1073: Move all journal stream management code into one place
Date Wed, 24 Aug 2011 23:09:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090605#comment-13090605
] 

Eli Collins commented on HDFS-2018:
-----------------------------------

Here are the notes from the call we had today:
# We discussed whether to have an explicit recoverLogs API in JournalManager. Todd explained
the motivation for the API - shared storage the standy wants to call getInputStream w/o necesarily
initiating recovery, so the explicit API is needed.
# We discussed whether/how to expose the segments. Jitendra discussed one option: hiding the
segments via having a JournalAdminInterface with getEditLogManifest method that manages transfer
from primary to the standby and checkpointer. The JournalManager and the checkpointer both
implement this interface. Todd discussed an intermediate step, using the tuples of log segments
and delaying JournalAdminInterface to 1580. Sanjay proposed trying to get rid of segments
in 1580 and doing that incrementally in a separate jira, people thought that was reasonable.
# Todd raised some concerns about removing the caching introduced in HDFS-2018, Sanjay and
Jitendra are going to take a look and comment.


> 1073: Move all journal stream management code into one place
> ------------------------------------------------------------
>
>                 Key: HDFS-2018
>                 URL: https://issues.apache.org/jira/browse/HDFS-2018
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
hdfs-2018-otherapi.txt, hdfs-2018.txt
>
>
> Currently in the HDFS-1073 branch, the code for creating output streams is in FileJournalManager
and the code for input streams is in the inspectors. This change does a number of things.
>   - Input and Output streams are now created by the JournalManager.
>   - FSImageStorageInspectors now deals with URIs when referring to edit logs
>   - Recovery of inprogress logs is performed by counting the number of transactions instead
of looking at the length of the file.
> The patch for this applies on top of the HDFS-1073 branch + HDFS-2003 patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message