Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Sat, 3 May 2014 10:54:15 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12711167.1398776473204.225129.1399114455226@arcas>
In-Reply-To: <JIRA.12711167.1398776473204@arcas>
References: <JIRA.12711167.1398776473204@arcas>
Subject: [jira] [Comment Edited] (HDFS-6301) NameNode: persist XAttrs in
 fsimage and record XAttrs modifications to edit log.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988644#comment-13988644 ] 

Uma Maheswara Rao G edited comment on HDFS-6301 at 5/3/14 10:53 AM:
--------------------------------------------------------------------

{quote}
Why do we need to modify OP_ADD, and have xattrs be an argument to unprotectedAddFile? AFAIK new files don't have any xattrs until they're set. The other use of OP_ADD is for append, which also does not involve setting xattrs. This looks like c+p from the ACL code, which needs it because of default ACLs.
{quote}
Yes, you are right that we will not have any xattrs on file/dir creation time.

{quote}
 I know for one that DFSTestUtil#runOperations needs to be updated with the new op, and the test resources related to TestOfflineEditsViewer will need to be updated. These are fairly mechanical and we can do it in a follow-on if you like.
{quote}
+1 to do on followup JIRAs


Thinking on OP_SET_XATTRS op code, 
 When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and let's say [USER.name1:value1]
 On a next call if we set another xattrs, then it may store along with older existing xattrs again. It may be like [USER.name1:value1, USER.name2:value2]
 So, on adding more xattrs on same Inode, that list may grow and we keep store the entries of already existing name, value fairs.
 
 Right now we defaulted the max Xattrs on an Inode to 32 and configured. If user modified it to much larger value and start setting xattrs, then edits loading may create issue like my above example.
 But I didn't refer any usecase of having large number of xattrs, this is just the assumption to consider a case. My biggest doubt is whether we will have such real usecases to have huge xattrs on a single INode. 

 So, here is a thought on having OP_SET_XATTR for each setXAttr operation to be logged, When we replay them we need to consolidate. This is some initial thought we can think more if others also feel we need to consider this case to handle.
 But we don't need to block the current JIRA, we can do it in followup JIRA as well.


was (Author: umamaheswararao):
{quote}
Why do we need to modify OP_ADD, and have xattrs be an argument to unprotectedAddFile? AFAIK new files don't have any xattrs until they're set. The other use of OP_ADD is for append, which also does not involve setting xattrs. This looks like c+p from the ACL code, which needs it because of default ACLs.
{quote}
Yes, you are right that we will not have any xattrs on file/dir creation time.

{quote}
 I know for one that DFSTestUtil#runOperations needs to be updated with the new op, and the test resources related to TestOfflineEditsViewer will need to be updated. These are fairly mechanical and we can do it in a follow-on if you like.
{quote}
+1 to do on followup JIRAs


Thinking on OP_SET_XATTRS op code, 
 When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and let's say [USER.name1:value1]
 On a next call if we set another xattrs, then it may store along with older existing xattrs again. It may be like [USER.name1:value1, USER.name2:value2]
 So, on adding more xattrs on same Inode, that list may grow and we keep store the entries of already existing name, value fairs.
 
 Right now we defaulted the max Xattrs on an Inode to 32 and configured. If user modified it to much larger value and start setting xattrs, then edits loading may create issue like my above example.
 But I didn't refer any usecase of having large number of xattrs, this is just the assumption to consider a case. 

 So, here is a thought on having OP_SET_XATTR for each setXAttr operation to be logged, When we replay them we need to consolidate. This is some initial thought we can think more if others also feel we need to consider this case to handle.
 But we don't need to block the current JIRA, we can do it in followup JIRA as well.

> NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log.
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-6301
>                 URL: https://issues.apache.org/jira/browse/HDFS-6301
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: HDFS XAttrs (HDFS-2006)
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>             Fix For: HDFS XAttrs (HDFS-2006)
>
>         Attachments: HDFS-6301.patch
>
>
> Store XAttrs in fsimage so that XAttrs are retained across NameNode restarts.
> Implement a new edit log opcode, {{OP_SET_XATTRS}}.


--
This message was sent by Atlassian JIRA
(v6.2#6252)