hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: modular build and pluggable rpc
Date Fri, 27 May 2011 20:30:09 GMT
Agreed - I'm all for Thrift.

Though, I actually, contrary to Ryan, think that the existing HBaseRPC
handler/client code is pretty good -- better than the equivalents from
Thrift Java.

We could start by using Thrift serialization on our existing transport
-- then maybe work towards contributing it upstream to the Thrift
project. HDFS folks are potentially interested in doing that as well.


On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> I'm -1 on avro as a RPC format.  Thrift is the way to go, any of the
> advantages of smaller serialization of avro is lost by the sheer
> complexity of avro and therefore the potential bugs.
> I understand the desire to have a pluggable RPC engine, but it feels
> like the better approach would be to adopt a unified RPC and just be
> done with it.  I had a look at the HsHa mechanism in thrift and it is
> very good, it in fact matches our 'handler' approach - async
> recieving/sending of data, but single threaded for processing a
> message.
> -ryan
> On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <apurtell@apache.org> wrote:
>> Also needing, perhaps later, consideration:
>> - HDFS-347 or not
>>  - Lucene embedding for hbase-search, though as a coprocessor this is already pretty
much handled if we have platform support (therefore a platform module) for a HDFS that can
do local read shortcutting and block placement requests
>> - HFile v1 versus v2
>> Making decoupled development at several downstream sites manageable, with a home
upstream for all the work, while simultaneously providing clean migration paths for users,
>> --- On Fri, 5/27/11, Andrew Purtell <apurtell@apache.org> wrote:
>>> From: Andrew Purtell <apurtell@apache.org>
>>> Subject: modular build and pluggable rpc
>>> To: dev@hbase.apache.org
>>> Date: Friday, May 27, 2011, 12:49 PM
>>> From IRC:
>>> apurtell    i propose we take the build modular as early as possible to deal
with multiple platform targets
>>> apurtell    secure vs nonsecure
>>> apurtell    0.20 vs 0.22 vs trunk
>>> apurtell    i understand the maintenence issues with multiple rpc engines,
for example, but a lot of reflection twistiness is going to be worse
>>> apurtell    i propose we take up esammer on his offer
>>> apurtell    so branch 0.92 asap, get trunk modular and working against multiple
platform targets
>>> apurtell    especially if we're going to see rpc changes coming from downstream
>>> apurtell    also what about supporting secure and nonsecure clients with the
same deployment?
>>> apurtell    zookeeper does this
>>> apurtell    so that is selectable rpc engine per connection, with a negotiation
>>> apurtell    we don't have or want to be crazy about it but a rolling upgrade
should be possible if for example we are taking in a new rpc from fb (?) or cloudera (avro
>>> apurtell    also looks like hlog modules for 0.20 vs 0.22 and successors
>>> apurtell    i think over time we can roadmap the rpc engines, if we have multiple,
by deprecation
>>> apurtell    now that we're on the edge of supporting both 0.20 and 0.22, and
secure vs nonsecure, let's get it as manageable as possible right away
>>> St^Ack_        apurtell: +1
>>> apurtell    also i think there is some interest in async rpc engine
>>> St^Ack_        we should stick this up on dev i'd say
>>> Best regards,
>>>     - Andy
>>> Problems worthy of attack prove their worth by hitting
>>> back. - Piet Hein (via Tom White)

Todd Lipcon
Software Engineer, Cloudera

View raw message