hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
Date Thu, 09 Sep 2021 05:44:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=648395&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648395
]

ASF GitHub Bot logged work on HDFS-16187:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 09/Sep/21 05:43
            Start Date: 09/Sep/21 05:43
    Worklog Time Spent: 10m 
      Work Description: bshashikant commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-915780582


   > @bshashikant , thank you for the patch.
   > 
   > I see that the patch works by converting logic from reference-equals to value-equals.
In `AclStorage`, we maintain a reference-counted mapping of every immutable instance of every
`AclFeature` currently in use by the namesystem. Inodes with identical ACL entries all point
to the same `AclFeature`, so reference equals should work fine. This was an intentional choice
of the ACL design to lower memory footprint and provide an inexpensive equality operation.
   > 
   > Xattrs do not use this same flyweight pattern though, so I can see why deep equals
would be necessary there. I wonder if this bug really only happens for xattrs and not ACLs?
If the bug happens for ACLs, then that might mean there is really something wrong with reference
counting in `AclStorage`, causing duplicate instances for the same ACL entries.
   > 
   > Also, if value equals is necessary, then there is similar logic in `INodeFile` and
`INodeFileAttributes`.
   
   Thanks @cnauroth for the explanation. I tested the acl feature and it seems the issue only
seem to exist with Xattr feature not acl itself as you suggested. I have modified the patch
accordingly. Thank you very much for explanation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 648395)
    Time Spent: 1.5h  (was: 1h 20m)

> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with
checkpointing
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-16187
>                 URL: https://issues.apache.org/jira/browse/HDFS-16187
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: snapshots
>            Reporter: Srinivasu Majeti
>            Assignee: Shashikant Banerjee
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not consistent with
Xattr(EZ here settinh the Xattr) across NN restarts with checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
>   final Path snapshottable = new Path("/zones");
>   fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
>       true);
>   dfsAdmin.allowSnapshot(snapshottable);
>   dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
>   fs.createSnapshot(snapshottable, "snap1");
>   SnapshotDiffReport report =
>       fs.getSnapshotDiffReport(snapshottable, "snap1", "");
>   Assert.assertEquals(0, report.getDiffList().size());
>   report =
>       fs.getSnapshotDiffReport(snapshottable, "snap1", "");
>   System.out.println(report);
>   Assert.assertEquals(0, report.getDiffList().size());
>   fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
>   fs.saveNamespace();
>   fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
>   cluster.restartNameNode(true);
>   report =
>       fs.getSnapshotDiffReport(snapshottable, "snap1", "");
>   Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail with below
error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot xxxxx
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message