hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: modular build and pluggable rpc
Date Fri, 27 May 2011 21:38:34 GMT
I don't disagree with any of this but the fact is we have compile time differences if going
against secure Hadoop 0.20 or non-secure Hadoop 0.20. 

So either we decide to punt on integration with secure Hadoop 0.20 or we deal with the compile
time differences. If dealing with them, we can do it by reflection, which is brittle and can
be difficult to understand and debug, and someone would have to do this work; or we can wholesale
replace RPC with something based on Thrift, and someone would have to do the work; or we take
the pluggable RPC changes that Gary has already developed and modularize the build, which
Eric has already volunteered to do.

  - Andy

--- On Fri, 5/27/11, Todd Lipcon <todd@cloudera.com> wrote:

> From: Todd Lipcon <todd@cloudera.com>
> Subject: Re: modular build and pluggable rpc
> To: dev@hbase.apache.org
> Cc: apurtell@apache.org
> Date: Friday, May 27, 2011, 1:30 PM
> Agreed - I'm all for Thrift.
> 
> Though, I actually, contrary to Ryan, think that the
> existing HBaseRPC
> handler/client code is pretty good -- better than the
> equivalents from
> Thrift Java.
> 
> We could start by using Thrift serialization on our
> existing transport
> -- then maybe work towards contributing it upstream to the
> Thrift
> project. HDFS folks are potentially interested in doing
> that as well.
> 
> -Todd
> 
> On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <ryanobjc@gmail.com>
> wrote:
> > I'm -1 on avro as a RPC format.  Thrift is the way to
> go, any of the
> > advantages of smaller serialization of avro is lost by
> the sheer
> > complexity of avro and therefore the potential bugs.
> >
> > I understand the desire to have a pluggable RPC
> engine, but it feels
> > like the better approach would be to adopt a unified
> RPC and just be
> > done with it.  I had a look at the HsHa mechanism in
> thrift and it is
> > very good, it in fact matches our 'handler' approach -
> async
> > recieving/sending of data, but single threaded for
> processing a
> > message.
> >
> > -ryan
> >
> > On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <apurtell@apache.org>
> wrote:
> >> Also needing, perhaps later, consideration:
> >>
> >> - HDFS-347 or not
> >>
> >>  - Lucene embedding for hbase-search, though as a
> coprocessor this is already pretty much handled if we have
> platform support (therefore a platform module) for a HDFS
> that can do local read shortcutting and block placement
> requests
> >>
> >> - HFile v1 versus v2
> >>
> >> Making decoupled development at several downstream
> sites manageable, with a home upstream for all the work,
> while simultaneously providing clean migration paths for
> users, basically.
> >>
> >> --- On Fri, 5/27/11, Andrew Purtell <apurtell@apache.org>
> wrote:
> >>
> >>> From: Andrew Purtell <apurtell@apache.org>
> >>> Subject: modular build and pluggable rpc
> >>> To: dev@hbase.apache.org
> >>> Date: Friday, May 27, 2011, 12:49 PM
> >>> From IRC:
> >>>
> >>> apurtell    i propose we take the build
> modular as early as possible to deal with multiple platform
> targets
> >>> apurtell    secure vs nonsecure
> >>> apurtell    0.20 vs 0.22 vs trunk
> >>> apurtell    i understand the maintenence
> issues with multiple rpc engines, for example, but a lot of
> reflection twistiness is going to be worse
> >>> apurtell    i propose we take up esammer on
> his offer
> >>> apurtell    so branch 0.92 asap, get trunk
> modular and working against multiple platform targets
> >>> apurtell    especially if we're going to
> see rpc changes coming from downstream projects...
> >>> apurtell    also what about supporting
> secure and nonsecure clients with the same deployment?
> >>> apurtell    zookeeper does this
> >>> apurtell    so that is selectable rpc
> engine per connection, with a negotiation
> >>> apurtell    we don't have or want to be
> crazy about it but a rolling upgrade should be possible if
> for example we are taking in a new rpc from fb (?) or
> cloudera (avro based?)
> >>> apurtell    also looks like hlog modules
> for 0.20 vs 0.22 and successors
> >>> apurtell    i think over time we can
> roadmap the rpc engines, if we have multiple, by
> deprecation
> >>> apurtell    now that we're on the edge of
> supporting both 0.20 and 0.22, and secure vs nonsecure,
> let's get it as manageable as possible right away
> >>>
> >>> St^Ack_        apurtell: +1
> >>>
> >>> apurtell    also i think there is some
> interest in async rpc engine
> >>>
> >>> St^Ack_        we should stick this up
> on dev i'd say
> >>>
> >>> Best regards,
> >>>
> >>>     - Andy
> >>>
> >>> Problems worthy of attack prove their worth by
> hitting
> >>> back. - Piet Hein (via Tom White)
> >>>
> >>
> >
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera
> 

Mime
View raw message