hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Proposal: abandon protocol translator layer for cluster-internal RPCs
Date Mon, 19 Mar 2012 15:55:29 GMT
Exactly. We still have wire compatibility, we just avoid the redundant
work of wrapping the whole thing.


On Mon, Mar 19, 2012 at 8:11 AM, Robert Evans <evans@yahoo-inc.com> wrote:
> I think what we are talking about here is removing some of the extra layers of abstraction
in java.  The wire protocol used will be identical in either case.  It is just that we would
have to use the Protocol Buffer Builder APIs instead of wrapping them with our own custom
> I am +1 for reducing the layers needed to add/modify the RPC.
> --Bobby Evans
> On 3/19/12 2:17 AM, "Jitendra Pandey" <jitendra@hortonworks.com> wrote:
>  Wire compatibility in hdfs private protocols between different
> components is also important for rolling upgrades. We do want to support
> upgrading different components of a cluster independent of each other and
> wire compatibility is one of the essential prerequisites. Therefore, even
> if some protocols are not exposed to the users, and are just used within
> different internal components, we still cannot afford to compromise on wire
> compatibility on those interfaces.
> On Sat, Mar 17, 2012 at 3:31 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> Hi all,
>> I've been working on some patches recently that required adding a new
>> protocol and some RPC calls that will be used entirely internally to
>> HDFS -- i.e the types and functions are never exposed to clients. The
>> process to do this involved:
>> 1) Add a new .proto file MyProtocol.proto with the types and the
>> service definition
>> 2) Add a new empty Java interface MyProtocolPB.java which adds the
>> ProtocolInfo and KerberosInfo annotations
>> 3) Add a new Java interface MyProtocol.java which duplicates the same
>> methods I defined in the protobuf service
>> 4) For each new type, create a new Java class which duplicates the
>> fields, getters, and setters from the protobuf messages
>> 5) Create a Client-Side Translator and Server-Side Translator class,
>> each containing a wrapper method for each of the calls
>> 6) Create a PBHelper class which contains two convert() methods for
>> each of the new types
>> Given that we have many protocols that we never intend to expose, I
>> see little benefit to adding the indirection layer here. It only makes
>> the task of modifying the protocols quite onerous and full of
>> duplicate boilerplate code.
>> I'd like to propose that, when adding protocols that are meant to be
>> HDFS-private, we drop steps 3-6 and use the protobuf RPC engine
>> directly. Doing this doesn't force our hand or limit our options in
>> the future -- should we want to add an alternate mechanism one can
>> always add the indirection layer down the road.
>> Thoughts?
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
> --

Todd Lipcon
Software Engineer, Cloudera

View raw message