hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
Date Fri, 04 Feb 2011 21:39:30 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990760#comment-12990760

Jitendra Nath Pandey commented on HDFS-1580:

> > ShouldWALStreamCustodians control rolling themselves?
  As I understand, book-keeper doesn't allow reading logs from open ledger, is that correct?
If that is the case it would be better to first roll and then do the checkpoint to capture
most latest edits. 

  It seems to me this will be implemented only by fsedit log. If that is the case we could
handle errors just by exceptions i.e. if an operation on WALStreamCustodian interface fails
an exception is thrown and fsedit log can decide to remove the custodian depending on the
kind of exception.

> namenode crash
  If the namenode comes back before the znode disappears from zookeeper, the ledger will be
open. In that case, will the FSEditLog.load be able to load transactions from the open ledger
as well?

> interface JournalStream
  The document defines this interface but doesn't describe its purpose or use-case.

> List<URI> getLogs(long sinceTransactionId);
   The list returned must be ordered w.r.t the transactions contained. It might be a good
idea to encode the ordering attribute in the url itself, so that the caller of this method
can also verify that order is correct. The uri naming convention could mimic this aspect from
the convention proposed in 1073.

> void startRoll()
> void endRoll()
    I can only imagine a single roll method, that cuts a log, and starts a new one. I believe
the naming convention or the ordering attribute for the logs should be controlled by the application
and not the storage, therefore the roll method should take a parameter which becomes part
of the log metadata and is used to order the logs. Again this also depends on how 1073 does
it for file logs.

> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>                 Key: HDFS-1580
>                 URL: https://issues.apache.org/jira/browse/HDFS-1580
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>         Attachments: generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message