hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1521) Persist transaction ID on disk between NN restarts
Date Tue, 21 Dec 2010 10:18:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973543#action_12973543
] 

Konstantin Shvachko commented on HDFS-1521:
-------------------------------------------

2. I started thinking why do we need {{FSEditLogLoader.needResave}}. First of all it is only
one of the conditions that requires resave. Second, it is relevant only if the edits is not
empty, otherwise it is going to be saved based on the transaction count. So the name should
be {{isOlderVersion}} instead of {{needResave}}, which led me to the idea that we don't need
to read edits to know it's an old version as we already know that after reading VERSION file
even before we loaded fsimage. 
So we can remove {{FSEditLogLoader.needResave}} along with all the logic related to it from
EditsLog, and move this logic into {{FSImage.loadFSImage(File)}}
{code}
  boolean loadFSImage(File curFile) throws IOException {
    ..............
    boolean needToSave = this.layoutVersion != FSConstants.LAYOUT_VERSION;
    return needToSave;
  }
{code}
It should {{this.layoutVersion}} not the loaded one as described below.

Then I noticed that we read layoutVersion and namespaceId from fsimage file and replace with
them values obtained from  the VERSION file. We shouldn't be doing that. This is not introduced
in your patch. It has been there for a while. But we should have fixed it during refactoring.
I should have looked at the TODO comment that I myself wrote there. I think we need fix it
now.
So the idea is that we should never rely on the values for namespaceId and layoutVersion read
from fsimage. We should read them for backward compatibility, but then completely ignore them.
Therefore we will not need those 2 fields
{{FSImageFormat.Loader.imgVersion}}
{{FSImageFormat.Loader.imgNamespaceID}}
and related getters, as we will not need to assign those values in FSImage.
If you think it's a different issue I can file one. But {{needResave}} does not need to be
introduced here based on the above.

Finally, this gets me to the question if {{FSImage.checkpointTxId}} as you describe it should
be written to fsimage or to the VERSION file. Feels like VERSION is the right place. For image
directories it is the latest txId committed to this image. For edits-only directories it is
the {{expectedStartingTxId - 1}}.

10. Yes you can surround a call to rollEditLog() and getLastWrittenTxId() with any synchronized
sections you need.

11. I do not understand how {{lastAppliedTxId}} is initialized teh first time when BN starts
up. May be BN should not count txIds but rather receive startTxId from NN.

> Persist transaction ID on disk between NN restarts
> --------------------------------------------------
>
>                 Key: HDFS-1521
>                 URL: https://issues.apache.org/jira/browse/HDFS-1521
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hdfs-1521.3.txt, hdfs-1521.4.txt, hdfs-1521.5.txt, hdfs-1521.txt,
hdfs-1521.txt
>
>
> For HDFS-1073 and other future work, we'd like to have the concept of a transaction ID
that is persisted on disk with the image/edits. We already have this concept in the NameNode
but it resets to 0 on restart. We can also use this txid to replace the _checkpointTime_ field,
I believe.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message