hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakuma...@huawei.com>
Subject RE: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server
Date Thu, 20 Mar 2014 04:25:27 GMT
+1, I agree with your point Chris. It depends on the client application how they using the
hdfs jars in their classpath.

As implementation already supports the compatibility (through protobuf), No extra code changes
required to support new Client + old server.

I feel it will be good to explicitly mention about the compatibility of existing APIs in both
versions.

Anyway this is not applicable for the new APIs in latest client and this is understood. We
can make it explicit in the document though.


Regards,
Vinayakumar B

-----Original Message-----
From: Chris Nauroth [mailto:cnauroth@hortonworks.com] 
Sent: 20 March 2014 05:36
To: common-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

I think this kind of compatibility issue still could surface for HDFS, particularly for custom
applications (i.e. something not executed via "hadoop jar" on a cluster node, where the client
classes ought to be injected into the classpath automatically).  Running DistCP between 2
clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client as a dependency
and try to use it to make HTTP calls to a 2.3.0 HDFS cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of 
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cnauroth@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> /Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue 
> > using the old clients even after upgrading the server (cluster) to a 
> > later version (or vice versa). For example, a Hadoop 2.1.0 client 
> > talking to a Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions 
> > within an active cluster so the cluster may be upgraded without 
> > downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client 
> > ahead of the server.  (There is no clause for "upgraded client + old 
> > server".) Based on my experience, this is a valid use case when a 
> > user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's 
> > upgrade schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the 
> > answer
> is
> > yes and coded our new features accordingly, but it isn't made 
> > explicit in the documentation.  Do we all agree that the answer is 
> > yes, or is it possibly up for debate depending on the change in 
> > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way, 
> > I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved 
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a 
> > configuration option.  By default, it will continue using the 
> > existing syntax currently understood by the pre-2.4.0 NodeManager, 
> > thus preserving compatibility.  We wanted to open the policy 
> > question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or 
> > entity
> to
> > which it is addressed and may contain information that is 
> > confidential, privileged and exempt from disclosure under applicable 
> > law. If the reader of this message is not the intended recipient, 
> > you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or 
> > forwarding of this communication is strictly prohibited. If you have 
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or 
> entity to which it is addressed and may contain information that is 
> confidential, privileged and exempt from disclosure under applicable 
> law. If the reader of this message is not the intended recipient, you 
> are hereby notified that any printing, copying, dissemination, 
> distribution, disclosure or forwarding of this communication is 
> strictly prohibited. If you have received this communication in error, 
> please contact the sender immediately and delete it from your system. Thank You.
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.

Mime
View raw message