hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: RPC versioning
Date Tue, 07 Oct 2008 13:30:42 GMT
Alejandro Abdelnur wrote:
>> ....
>> Question 1: Do we need to solve this problem soon, for release 1.0, i.e., in
>> order to provide a release whose compatibility lifetime is ~1 year, instead
>> of the ~4months of 0. releases?  This is not clear to me.  Can someone
>> provide cases where using the same classpath when talking to multiple
>> clusters is critical?
> 
> Yes, we have this requirement to be able to do upgrades of our system
> without downtime.
> 
> Our hadoop jobs are managed from an appserver. Our appserver can
> dispatch jobs to different hadoop clusters. To do a hadoop upgrade
> without downtime we would redirect jobs out of one of the hadoop
> clusters (of course you have to ge the necessary INPUTs out too),
> upgrade that idle cluster and then redirect traffic back. At this
> point our server will be running vN while the upgraded cluster will be
> runing vN+1. Then we can later upgrade our appserver to vN+1 by taking
> instances out of rotation one at the time.
> 
> Note that the situation of the appserver interacting with vN and vN+1
> hadoop clusters could be for a significant period of time (weeks if
> not months).
> 
>> ....
>> Below I suggest a simple versioning style that Hadoop might use to
>> permit its RPC protocols to evolve compatibly ...
>> ...
>> When an RPC client and server handshake, they exchange version
>> numbers.  The lower of their two version numbers is selected as the
>> version for the connection.
> 
> How about something more simplistic?
> 
> * The client handles a single version of Hadoop, the one that it belongs
> * The server handles its version and a a backwards range [vN, vN-M]
> * All client RPCs always start with the Hadoop version they belong to
> * On the server side an RpcSwitch reads the version, checks the
> received RPC is within valid version range, and delegates the call to
> corresponding handler. The vN-1 to vN-M handlers will be adapters to a
> vN handler

I'm going to make another suggestion, which is : are there 
long-haul/high stability APIs which we want to offer alongside the 
unstable/higher performance IPC protocols. So IPC could be used within 
the cluster, but something else would run for talking to the cluster 
from the outside
  -assumes no trust of the caller and requires rigorous authentication
  -less brittle to changes in the serialization formats
  -designed with firewalls and timeouts in mind
A few years back, people would have said SOAP! or SOA, but something 
that uses JSON over REST is the other option. Push up the files using 
WebDAV, then use a RESTy interface to the JobTrackers to submit work

There's engineering effort here, but this would be the API used by 
things like IDE clients, and other applications to get work done

Mime
View raw message