hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7056) Snapshot support for truncate
Date Tue, 23 Dec 2014 00:44:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256374#comment-14256374
] 

Jing Zhao commented on HDFS-7056:
---------------------------------

Some further comments:
# In {{FileDiffList#combineAndCollectSnapshotBlocks}}, why do we pass {{null}} for {{collectBlocksAndClear}}?
In case that the file has been deleted and we're deleting the last snapshot, we need to delete
the whole INode thus should pass a non-null INode List to {{collectBlocksAndClear}}. We should
also have unit tests for this if we can confirm this is an issue.
{code}
+    BlockInfo[] removedBlocks = removed.getBlocks();
+    if(removedBlocks == null) {
+      FileWithSnapshotFeature sf = file.getFileWithSnapshotFeature();
+      assert sf != null : "FileWithSnapshotFeature is null";
+      if(sf.isCurrentFileDeleted())
+        sf.collectBlocksAndClear(file, collectedBlocks, null);
+      return;
+    }
{code}
# The semantic of {{findEarlierSnapshotBlocks}} is not easy to follow. Looks like when {{snapshotId}}
is {{Snapshot.CURRENT_STATE_ID}} the function selects an exclusive semantic, while for others
the function chooses an inclusive semantic. We need to make it more consistent here. Also
it can be optimized similarly as {{findLaterSnapshotBlocks}}.
# Minor: in the current patch {{findLaterSnapshotBlocks}} is always coupled with an extra
null check and if it returns null {{getBlocks}} is called. This extra check/call can actually
be combined into {{findLaterSnapshotBlocks}}.
{code}
+    snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId());
+    return (snapshotBlocks == null) ? getBlocks() : snapshotBlocks;
{code}
# bq. Also bumped up the NameNode layout version.
Could you generally explain why we need to bump up the NN layout version here? The protobuf-based
fsimage should be able to handle its compatibility. One benefit for bumping up layoutversion
can be preventing rolling downgrade so that an fsimage produced by "snapshot+truncate" cannot
be directly loaded by the old version jar.

> Snapshot support for truncate
> -----------------------------
>
>                 Key: HDFS-7056
>                 URL: https://issues.apache.org/jira/browse/HDFS-7056
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Plamen Jeliazkov
>         Attachments: HDFS-3107-HDFS-7056-combined-13.patch, HDFS-3107-HDFS-7056-combined.patch,
HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch,
HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch,
HDFS-3107-HDFS-7056-combined.patch, HDFS-3107-HDFS-7056-combined.patch, HDFS-7056-13.patch,
HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch, HDFS-7056.patch,
HDFS-7056.patch, HDFS-7056.patch, HDFSSnapshotWithTruncateDesign.docx
>
>
> Implementation of truncate in HDFS-3107 does not allow truncating files which are in
a snapshot. It is desirable to be able to truncate and still keep the old file state of the
file in the snapshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message