hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur" <tuc...@gmail.com>
Subject Re: RPC versioning
Date Tue, 07 Oct 2008 10:48:36 GMT
> ....
> Question 1: Do we need to solve this problem soon, for release 1.0, i.e., in
> order to provide a release whose compatibility lifetime is ~1 year, instead
> of the ~4months of 0. releases?  This is not clear to me.  Can someone
> provide cases where using the same classpath when talking to multiple
> clusters is critical?

Yes, we have this requirement to be able to do upgrades of our system
without downtime.

Our hadoop jobs are managed from an appserver. Our appserver can
dispatch jobs to different hadoop clusters. To do a hadoop upgrade
without downtime we would redirect jobs out of one of the hadoop
clusters (of course you have to ge the necessary INPUTs out too),
upgrade that idle cluster and then redirect traffic back. At this
point our server will be running vN while the upgraded cluster will be
runing vN+1. Then we can later upgrade our appserver to vN+1 by taking
instances out of rotation one at the time.

Note that the situation of the appserver interacting with vN and vN+1
hadoop clusters could be for a significant period of time (weeks if
not months).

> ....
> Below I suggest a simple versioning style that Hadoop might use to
> permit its RPC protocols to evolve compatibly ...
> ...
> When an RPC client and server handshake, they exchange version
> numbers.  The lower of their two version numbers is selected as the
> version for the connection.

How about something more simplistic?

* The client handles a single version of Hadoop, the one that it belongs
* The server handles its version and a a backwards range [vN, vN-M]
* All client RPCs always start with the Hadoop version they belong to
* On the server side an RpcSwitch reads the version, checks the
received RPC is within valid version range, and delegates the call to
corresponding handler. The vN-1 to vN-M handlers will be adapters to a
vN handler


View raw message