hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
Date Tue, 22 Mar 2011 01:52:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009512#comment-13009512
] 

Jitendra Nath Pandey commented on HDFS-1580:
--------------------------------------------

A few comments.

1. > JournalManager#transfer()
  Is this a push method to be executed on primary or a pull from secondary? The mechanism
as we have today, which is more like a pull, can be provided by the interface using getLogs
and getInputStream methods, which, in my opinion, is sufficient.

2. We could use single BookKeeper installation (or any shared storage solution) for storing
edit logs from multiple namenodes (e.g. federation). In that case a secondary/backup should
be able to instantiate a JournalManager and request logs for a specific namenode. Similar
could be argued about the files on namenodes, where backup or secondary wants to specify the
namenode where it wants to download the files from. 
   As I understand the steps to get edit logs at the backup node or secondary would be
   - Instantiate a JournalManager
   - specify the source of edit logs. This could be specified in the JournalManager constructor.
In case of BK the source would be the set of ledgers for a specific namenode.
   - getLogs from sinceTransactionId. It returns a list of URIs.
   - For each uri get the input stream and read the logs.

3. For each edit log we need to store some metadata, particularly the LAYOUT_VERSION. I think
it would make sense to provide for that in JournalManager interface.

4. For edit logs do we really need upgrade, rollback or finalizeUpgrade? For edit log files,
it seems to me that all that matters is whether the namenode has right version to understand
them or load them, which can be determined from LAYOUT_VERSION.

5. I will recommend putting JournalManager and its derived classes in separate packages.

6. Please add javadoc comments on JournalManager class and its methods.



> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>
>                 Key: HDFS-1580
>                 URL: https://issues.apache.org/jira/browse/HDFS-1580
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ivan Kelly
>         Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.txt
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message