Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of suresh@hortonworks.com
 designates 209.85.216.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGB5D2azHCQ0CDLCspP7ti-OOrQi6F16_jrS5zcOvYh5ZbxU8A@mail.gmail.com>
References: 
 <6A0439AD4731794E8EB53669FA76B11B0F3D11BE@BGSMSX102.gar.corp.intel.com>
	<CAGB5D2b2kuN7TU+As_ccnc-rJ1bbVGUi1mGo+auN3b0Bad6bYA@mail.gmail.com>
	<CABCYYb9i500kT2e_VHnWiFpECgAUCB7J9Gk5oaDin9meXUEyeQ@mail.gmail.com>
	<31351181-D548-457D-83AC-E180A43A8537@hortonworks.com>
	<CAGB5D2azHCQ0CDLCspP7ti-OOrQi6F16_jrS5zcOvYh5ZbxU8A@mail.gmail.com>
Date: Mon, 19 May 2014 09:44:28 -0700
Message-ID: 
 <CADdVdVF6C72durxRwO8xcf+QdGG7GY70pKSvpeLrC4=HbNqVWA@mail.gmail.com>
Subject: Re: [Vote] Merge The HDFS XAttrs Feature Branch (HDFS-2006) to Trunk
From: Suresh Srinivas <suresh@hortonworks.com>
To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e0153864e0ce71304f9c37a8d

--089e0153864e0ce71304f9c37a8d
Content-Type: text/plain; charset=UTF-8

NN layout version bump should be fine. This means during rolling upgrade
downgrade from 2.5 NN to 2.4 NN will not be possible. Only rollback can be
supported.

HDFS-2006 adds new RPCs. The question is, does the new client that depends
on this functionality gracefully fallback when talking to old server? Also
are there any incompatibilities between NN and DN (my guess is there is
none, given this is NN feature)?


On Fri, May 16, 2014 at 5:09 PM, Andrew Wang <andrew.wang@cloudera.com>wrote:

> Hi Suresh,
>
> HDFS-2006 adds new edit log ops (no modifications to existing ops) and a
> new XAttr INode Feature needs to be serialized to the fsimage. This
> required doing a NN layout version bump, so a 2.4 NN will not be able to
> start with these edit logs or images. Thus it's incompatible.
>
> I think you're better equipped to comment on rolling upgrades, but based on
> my understanding, I think the above means we should still be able to
> rolling upgrade between 2.4 and whatever version eventually includes xattr
> support.
>
> RPC-wise, HDFS-2006 only adds new RPCs, so I don't think there are any
> concerns there.
>
> Thanks,
> Andrew
>
>
> On Fri, May 16, 2014 at 2:56 PM, Suresh Srinivas <suresh@hortonworks.com
> >wrote:
>
> > I have not looked at the development closely. With rolling upgrades
> > feature support in, are there any incompatible changes with this feature?
> >
> > Sent from phone
> >
> > > On May 16, 2014, at 10:30 AM, Chris Nauroth <cnauroth@hortonworks.com>
> > wrote:
> > >
> > > +1 for the merge.
> > >
> > > I've participated in ongoing design discussions and code reviews on
> > > individual patches.  Yesterday, I completed a final review pass over
> the
> > > code in the feature branch, the design document, the end user
> > documentation
> > > and the test plan.  It looks ready to me.
> > >
> > > Thank you to everyone who made contributions on the feature branch.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > >
> > >
> > > On Wed, May 14, 2014 at 9:20 AM, Andrew Wang <andrew.wang@cloudera.com
> > >wrote:
> > >
> > >> +1 from me as well, I've participated in review and development of
> this
> > >> branch and think it's ready for merge.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>
> > >> On Wed, May 14, 2014 at 5:47 AM, Gangumalla, Uma
> > >> <uma.gangumalla@intel.com>wrote:
> > >>
> > >>> Hello HDFS Devs,
> > >>>  I would like to call for a vote to merge the HDFS Extended
> Attributes
> > >>> (XAttrs) feature from the HDFS-2006 branch to the trunk.
> > >>>  XAttrs are already widely supported on many operating systems,
> > >> including
> > >>> Linux, Windows, and Mac OS. This will allow storing attributes for
> HDFS
> > >>> file/directory.
> > >>>  XAttr consist of a name and a value and exist in one of 4
> namespaces:
> > >>> user, trusted, security, and system. An XAttr name is prefixed with
> one
> > >> of
> > >>> these namespaces, so for example, "user.myxattr".
> > >>>  Consistent with ongoing awareness of Namenode memory usage, the
> > maximum
> > >>> number and size of XAttrs on a file/directory are limited by a
> > >>> configuration parameter.
> > >>>  The design document contains more details and can be found here:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12644341/HDFS-XAttrs-Design-3.pdf
> > >>>  Development of this feature has been tracked in JIRA HDFS-2006:
> > >>> https://issues.apache.org/jira/browse/HDFS-2006
> > >>>  All of the development work for the feature is contained in the
> > >>> "HDFS-2006" branch:
> > >>> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2006
> > >>> As last tasks, we are working to support XAttrs via libhdfs, webhdfs
> as
> > >>> well as other minor improvements.
> > >>>  We intend to finish those enhancements before the vote completes and
> > >>> otherwise we could move them to top-level JIRAs as they can be
> tracked
> > >>> independently. User document is also ready for this feature.
> > >>>  Here the doc attached in JIRA:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12644787/ExtendedAttributes.html
> > >>> The XAttrs feature is backwards-compatible and enabled by default. A
> > >>> cluster administrator can disable it.
> > >>> Testing:
> > >>> We've developed more than 70 new tests which cover the XAttrs get,
> set
> > >>> and remove APIs through DistributedFileSystem and WebHdfsFileSystem,
> > the
> > >>> new XAttr CLI commands, HA, XAttr persistence in the fsimage and
> > related.
> > >>>  Additional  testing plans are documented in:
> > >>
> >
> https://issues.apache.org/jira/secure/attachment/12644342/Test-Plan-for-Extended-Attributes-1.pdf
> > >>>  Thanks a lot to the contributors who have helped and participated in
> > >> the
> > >>> branch development.
> > >>>  Code contributors are Yi Liu, Charles Lamb, Andrew Wang and Uma
> > >>> Maheswara Rao G.
> > >>> The design document incorporates feedback from many community
> members:
> > >>> Chris Nauroth, Andrew Purtell, Tianyou Li, Avik Dey, Charles Lamb,
> > >>> Alejandro, Andrew Wang, Tsz Wo Nicholas Sze and Uma Maheswara Rao G.
> > >>> Code reviewers on individual patches include Chris Nauroth,
> Alejandro,
> > >>> Andrew Wang, Charles Lamb, Tsz Wo Nicholas Sze and Uma Maheswara Rao
> G.
> > >>>
> > >>>  Also thanks to Dhruba for bringing up this JIRA and thanks to others
> > >> who
> > >>> participated for discussions.
> > >>> This vote will run for a week and close on 5/21/2014 at 06:16 pm IST.
> > >>>
> > >>> Here is my +1 to start with.
> > >>> Regards,
> > >>> Uma
> > >>> (umamahesh@apache.org<mailto:umamahesh@apache.org>)
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>


-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

--089e0153864e0ce71304f9c37a8d--