hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9696) Garbage snapshot records lingering forever
Date Fri, 12 Aug 2016 16:47:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419099#comment-15419099
] 

Kihwal Lee commented on HDFS-9696:
----------------------------------

Does something like this make sense?  Saving a diff section involves iterating the entire
inode map. When there is no snapshot, we can potentially cut down fsimage saving time and
reduce java object generation.
{code}
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
@@ -496,7 +496,10 @@ private void saveInternal(FileOutputStream fout,
       Step step = new Step(StepType.INODES, filePath);
       prog.beginStep(Phase.SAVING_CHECKPOINT, step);
       saveInodes(b);
-      saveSnapshots(b);
+      if (context.getSourceNamesystem().getSnapshotManager()
+          .getNumSnapshots() > 0) {
+        saveSnapshots(b);
+      }
       prog.endStep(Phase.SAVING_CHECKPOINT, step);
{code}

If no one objects, I will add a test case and submit a patch.

> Garbage snapshot records lingering forever
> ------------------------------------------
>
>                 Key: HDFS-9696
>                 URL: https://issues.apache.org/jira/browse/HDFS-9696
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.2
>            Reporter: Kihwal Lee
>            Priority: Critical
>
> We have a cluster where the snapshot feature might have been tested years ago. When the
HDFS does not have any snapshot, but I see filediff records persisted in its fsimage.  Since
it has been restarted many times and checkpointed over 100 times since then, it must haven
been persisted and  carried over since then.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message