Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates
 209.85.215.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADY20s43=Vdyn8rT9i=V+NCDWRpgkRLZr2Eh5naxagG0U1DhFA@mail.gmail.com>
References: 
 <CADY20s43=Vdyn8rT9i=V+NCDWRpgkRLZr2Eh5naxagG0U1DhFA@mail.gmail.com>
Date: Sun, 18 Mar 2012 19:49:00 -0700
Message-ID: 
 <CAPn_vTv3=wNg_UQNw0F10ofN792YfOjwx4Yvywc6R9mjh6yX5Q@mail.gmail.com>
Subject: Re: Proposal: abandon protocol translator layer for cluster-internal
 RPCs
From: Eli Collins <eli@cloudera.com>
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Sat, Mar 17, 2012 at 3:31 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi all,
>
> I've been working on some patches recently that required adding a new
> protocol and some RPC calls that will be used entirely internally to
> HDFS -- i.e the types and functions are never exposed to clients. The
> process to do this involved:
> 1) Add a new .proto file MyProtocol.proto with the types and the
> service definition
> 2) Add a new empty Java interface MyProtocolPB.java which adds the
> ProtocolInfo and KerberosInfo annotations
> 3) Add a new Java interface MyProtocol.java which duplicates the same
> methods I defined in the protobuf service
> 4) For each new type, create a new Java class which duplicates the
> fields, getters, and setters from the protobuf messages
> 5) Create a Client-Side Translator and Server-Side Translator class,
> each containing a wrapper method for each of the calls
> 6) Create a PBHelper class which contains two convert() methods for
> each of the new types
>
> Given that we have many protocols that we never intend to expose, I
> see little benefit to adding the indirection layer here. It only makes
> the task of modifying the protocols quite onerous and full of
> duplicate boilerplate code.
> I'd like to propose that, when adding protocols that are meant to be
> HDFS-private, we drop steps 3-6 and use the protobuf RPC engine
> directly. Doing this doesn't force our hand or limit our options in
> the future -- should we want to add an alternate mechanism one can
> always add the indirection layer down the road.
>
> Thoughts?
>

Sounds good to me.

I'm not sure what the indirection layer really buys us even for the
client <-> server protocols since just PB should be sufficient there.
But that's a separate discussion.


> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera