hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Proposal: abandon protocol translator layer for cluster-internal RPCs
Date Sat, 17 Mar 2012 22:31:57 GMT
Hi all,

I've been working on some patches recently that required adding a new
protocol and some RPC calls that will be used entirely internally to
HDFS -- i.e the types and functions are never exposed to clients. The
process to do this involved:
1) Add a new .proto file MyProtocol.proto with the types and the
service definition
2) Add a new empty Java interface MyProtocolPB.java which adds the
ProtocolInfo and KerberosInfo annotations
3) Add a new Java interface MyProtocol.java which duplicates the same
methods I defined in the protobuf service
4) For each new type, create a new Java class which duplicates the
fields, getters, and setters from the protobuf messages
5) Create a Client-Side Translator and Server-Side Translator class,
each containing a wrapper method for each of the calls
6) Create a PBHelper class which contains two convert() methods for
each of the new types

Given that we have many protocols that we never intend to expose, I
see little benefit to adding the indirection layer here. It only makes
the task of modifying the protocols quite onerous and full of
duplicate boilerplate code.
I'd like to propose that, when adding protocols that are meant to be
HDFS-private, we drop steps 3-6 and use the protobuf RPC engine
directly. Doing this doesn't force our hand or limit our options in
the future -- should we want to add an alternate mechanism one can
always add the indirection layer down the road.


Todd Lipcon
Software Engineer, Cloudera

View raw message