hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
Date Sat, 17 May 2014 00:09:06 GMT
Hi Suresh,

HDFS-2006 adds new edit log ops (no modifications to existing ops) and a
new XAttr INode Feature needs to be serialized to the fsimage. This
required doing a NN layout version bump, so a 2.4 NN will not be able to
start with these edit logs or images. Thus it's incompatible.

I think you're better equipped to comment on rolling upgrades, but based on
my understanding, I think the above means we should still be able to
rolling upgrade between 2.4 and whatever version eventually includes xattr
support.

RPC-wise, HDFS-2006 only adds new RPCs, so I don't think there are any
concerns there.

Thanks,
Andrew


On Fri, May 16, 2014 at 2:56 PM, Suresh Srinivas <suresh@hortonworks.com>wrote:

> I have not looked at the development closely. With rolling upgrades
> feature support in, are there any incompatible changes with this feature?
>
> Sent from phone
>
> > On May 16, 2014, at 10:30 AM, Chris Nauroth <cnauroth@hortonworks.com>
> wrote:
> >
> > +1 for the merge.
> >
> > I've participated in ongoing design discussions and code reviews on
> > individual patches.  Yesterday, I completed a final review pass over the
> > code in the feature branch, the design document, the end user
> documentation
> > and the test plan.  It looks ready to me.
> >
> > Thank you to everyone who made contributions on the feature branch.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <andrew.wang@cloudera.com
> >wrote:
> >
> >> +1 from me as well, I've participated in review and development of this
> >> branch and think it's ready for merge.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
> >> <uma.gangumalla@intel.com>wrote:
> >>
> >>> Hello HDFS Devs,
> >>>  I would like to call for a vote to merge the HDFS Extended Attributes
> >>> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> >>>  XAttrs are already widely supported on many operating systems,
> >> including
> >>> Linux, Windows, and Mac OS. This will allow storing attributes for HDFS
> >>> file/directory.
> >>>  XAttr consist of a name and a value and exist in one of 4 namespaces:
> >>> user, trusted, security, and system. An XAttr name is prefixed with one
> >> of
> >>> these namespaces, so for example, "user.myxattr".
> >>>  Consistent with ongoing awareness of Namenode memory usage, the
> maximum
> >>> number and size of XAttrs on a file/directory are limited by a
> >>> configuration parameter.
> >>>  The design document contains more details and can be found here:
> >>
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> >>>  Development of this feature has been tracked in JIRA HDFS-2006:
> >>> https://issues.apache.org/jira/browse/HDFS-2006
> >>>  All of the development work for the feature is contained in the
> >>> "HDFS-2006" branch:
> >>> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> >>> As last tasks, we are working to support XAttrs via libhdfs, webhdfs as
> >>> well as other minor improvements.
> >>>  We intend to finish those enhancements before the vote completes and
> >>> otherwise we could move them to top-level JIRAs as they can be tracked
> >>> independently. User document is also ready for this feature.
> >>>  Here the doc attached in JIRA:
> >>
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> >>> The XAttrs feature is backwards-compatible and enabled by default. A
> >>> cluster administrator can disable it.
> >>> Testing:
> >>> We've developed more than 70 new tests which cover the XAttrs get, set
> >>> and remove APIs through DistributedFileSystem and WebHdfsFileSystem,
> the
> >>> new XAttr CLI commands, HA, XAttr persistence in the fsimage and
> related.
> >>>  Additional  testing plans are documented in:
> >>
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> >>>  Thanks a lot to the contributors who have helped and participated in
> >> the
> >>> branch development.
> >>>  Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> >>> Maheswara Rao G.
> >>> The design document incorporates feedback from many community members:
> >>> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> >>> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> >>> Code reviewers on individual patches include Chris Nauroth, Alejandro,
> >>> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> >>>
> >>>  Also thanks to Dhruba for bringing up this JIRA and thanks to others
> >> who
> >>> participated for discussions.
> >>> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
> >>>
> >>> Here is my +1 to start with.
> >>> Regards,
> >>> Uma
> >>> (umamahesh@apache.org<mailto:umamahesh@apache.org>)
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message