hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3212) Persist the epoch received by the JournalService
Date Fri, 06 Apr 2012 00:28:23 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247894#comment-13247894
] 

Todd Lipcon commented on HDFS-3212:
-----------------------------------

bq. I do not understand what you mean by NN layer. Epoch is a notion from JournalManager to
the JournalNode. Both need to understand this and provide appropriate guarantees.

Currently, the NN code when starting a new log segment looks like this:
{code}
      editLogStream = journalSet.startLogSegment(segmentTxId);
...
    if (writeHeaderTxn) {
      logEdit(LogSegmentOp.getInstance(
          FSEditLogOpCodes.OP_START_LOG_SEGMENT));
      logSync();
    }
{code}

So the operation of starting a segment, and writing the OP_START_LOG_SEGMENT transaction are
separate. In general, the JournalManager abstraction doesn't know about the contents of the
edits it's writing -- it's just responsible for bytes. If you wanted to include the epoch
number in the OP_START_LOG_SEGMENT transaction, you'd have to have the NN code do something
like {{journalManager.getCurrentEpoch()}}, and then feed that into the logEdit call. But that's
not very generic, so it seems like a leak of abstraction.

bq. Whether you store it in a directory per-epoch or record it in the startlogSegment record
at the beginning of the segment - they are essentially the same.

I agree, if you're talking about prefixing it at the beginning of the file, before the first
transaction. But, if you're talking about actually putting it in the content of the first
transaction, I think it's a bad idea for the reason above. My preference is to keep it separated
from the file, so that the files written by JournalDaemon are exactly identical to the files
that would be written by FileJournalManager. That allows you to copy to and from the different
types of nodes without any difference in format.
                
> Persist the epoch received by the JournalService
> ------------------------------------------------
>
>                 Key: HDFS-3212
>                 URL: https://issues.apache.org/jira/browse/HDFS-3212
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>    Affects Versions: Shared journals (HDFS-3092)
>            Reporter: Suresh Srinivas
>
> epoch received over JournalProtocol should be persisted by JournalService.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message