hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-1521) Persist transaction ID on disk between NN restarts
Date Wed, 16 Feb 2011 18:23:24 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ivan Kelly updated HDFS-1521:
-----------------------------

    Attachment: HDFS-1521.diff

This patch addresses all Konstantin's comments except 11. There is something strange going
on with lastAppliedTxid, that TestBackupNode isn't currently picking up. At line 177 change
it to the following
{code}
      //
      // Take a checkpoint
      //
      backup = startBackupNode(conf, op, 1);
      waitCheckpointDone(backup);
      
      for (int i = 0; i < 10; i++) {
        writeFile(fileSys, new Path("file_" + i), replication);
      }
      
      backup.doCheckpoint();
      waitCheckpointDone(backup);

{code} 
This will trigger the test to fail. The normal run of the test doesn't exercise convergeJournalSpool,
so usually you don't see this. 

So, now you'll see that if BackupNode loads a checkpoint, and then tries to journal something,
the lastAppliedTxid + 1 will be 1 even though we've loaded in an image and editlog. The simple
fix is to put 
{code}
      lastAppliedTxId = getEditLog().getLastWrittenTxId();
{code} 
in loadCheckpoint(). This should be the end of the story.

However, with this change, you get the error 
{quote}
java.io.IOException: Expected transaction ID 10 but got 11
{quote}
A transaction is going missing. Whats happening is, when doCheckpoint get kicked off, the
log is rolled, and logJSpoolStart is called which creates an edit with opcode OP_JSPOOL_START.
This opcode, is caught by EditLogBackupOutputStream and never transmitted to the backup node,
so the transaction ids on the Primary and the Backup get out of sync. 

So, the question here is, is there any harm is actually transferring these OP_JSPOOL_START
transactions, or are they just excluded as a precaution?

> Persist transaction ID on disk between NN restarts
> --------------------------------------------------
>
>                 Key: HDFS-1521
>                 URL: https://issues.apache.org/jira/browse/HDFS-1521
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: HDFS-1521.diff, HDFS-1521.diff, hdfs-1521.3.txt, hdfs-1521.4.txt,
hdfs-1521.5.txt, hdfs-1521.txt, hdfs-1521.txt
>
>
> For HDFS-1073 and other future work, we'd like to have the concept of a transaction ID
that is persisted on disk with the image/edits. We already have this concept in the NameNode
but it resets to 0 on restart. We can also use this txid to replace the _checkpointTime_ field,
I believe.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message