hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Radia <sra...@yahoo-inc.com>
Subject Re: RPC versioning
Date Thu, 09 Oct 2008 22:55:11 GMT

On Oct 7, 2008, at 3:48 AM, Alejandro Abdelnur wrote:

> > ....
> > Question 1: Do we need to solve this problem soon, for release  
> 1.0, i.e., in
> > order to provide a release whose compatibility lifetime is ~1  
> year, instead
> > of the ~4months of 0. releases?  This is not clear to me.  Can  
> someone
> > provide cases where using the same classpath when talking to  
> multiple
> > clusters is critical?
> Yes, we have this requirement to be able to do upgrades of our system
> without downtime.
> Our hadoop jobs are managed from an appserver. Our appserver can
> dispatch jobs to different hadoop clusters. To do a hadoop upgrade
> without downtime we would redirect jobs out of one of the hadoop
> clusters (of course you have to ge the necessary INPUTs out too),
> upgrade that idle cluster and then redirect traffic back. At this
> point our server will be running vN while the upgraded cluster will be
> runing vN+1. Then we can later upgrade our appserver to vN+1 by taking
> instances out of rotation one at the time.
> Note that the situation of the appserver interacting with vN and vN+1
> hadoop clusters could be for a significant period of time (weeks if
> not months).
> > ....
> > Below I suggest a simple versioning style that Hadoop might use to
> > permit its RPC protocols to evolve compatibly ...
> > ...
> > When an RPC client and server handshake, they exchange version
> > numbers.  The lower of their two version numbers is selected as the
> > version for the connection.
> How about something more simplistic?
> * The client handles a single version of Hadoop, the one that it  
> belongs
> * The server handles its version and a a backwards range [vN, vN-M]
> * All client RPCs always start with the Hadoop version they belong to
> * On the server side an RpcSwitch reads the version, checks the
> received RPC is within valid version range, and delegates the call to
> corresponding handler. The vN-1 to vN-M handlers will be adapters to a
> vN handler
You will need to solve the following problem also:
Vn-1 Vn-2 ... each changed some field of some parameter class.
You can switch to the right handler but that handler in general only  
has a class definition of the class that matches its version. How does  
he read the objects with the older class definitions?
You may be able to address this via class loader tricks or by tagging  
a version number to the name of the class and keeping the definitions  
of each of the older versions (can be made a little simpler than I am  
describing though the magic of subclassing  ... but something like  
that will be needed.

> A

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message