hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: modular build and pluggable rpc
Date Fri, 27 May 2011 20:30:09 GMT
Agreed - I'm all for Thrift.

Though, I actually, contrary to Ryan, think that the existing HBaseRPC
handler/client code is pretty good -- better than the equivalents from
Thrift Java.

We could start by using Thrift serialization on our existing transport
-- then maybe work towards contributing it upstream to the Thrift
project. HDFS folks are potentially interested in doing that as well.

-Todd

On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> I'm -1 on avro as a RPC format.  Thrift is the way to go, any of the
> advantages of smaller serialization of avro is lost by the sheer
> complexity of avro and therefore the potential bugs.
>
> I understand the desire to have a pluggable RPC engine, but it feels
> like the better approach would be to adopt a unified RPC and just be
> done with it.  I had a look at the HsHa mechanism in thrift and it is
> very good, it in fact matches our 'handler' approach - async
> recieving/sending of data, but single threaded for processing a
> message.
>
> -ryan
>
> On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <apurtell@apache.org> wrote:
>> Also needing, perhaps later, consideration:
>>
>> - HDFS-347 or not
>>
>>  - Lucene embedding for hbase-search, though as a coprocessor this is already pretty
much handled if we have platform support (therefore a platform module) for a HDFS that can
do local read shortcutting and block placement requests
>>
>> - HFile v1 versus v2
>>
>> Making decoupled development at several downstream sites manageable, with a home
upstream for all the work, while simultaneously providing clean migration paths for users,
basically.
>>
>> --- On Fri, 5/27/11, Andrew Purtell <apurtell@apache.org> wrote:
>>
>>> From: Andrew Purtell <apurtell@apache.org>
>>> Subject: modular build and pluggable rpc
>>> To: dev@hbase.apache.org
>>> Date: Friday, May 27, 2011, 12:49 PM
>>> From IRC:
>>>
>>> apurtell    i propose we take the build modular as early as possible to deal
with multiple platform targets
>>> apurtell    secure vs nonsecure
>>> apurtell    0.20 vs 0.22 vs trunk
>>> apurtell    i understand the maintenence issues with multiple rpc engines,
for example, but a lot of reflection twistiness is going to be worse
>>> apurtell    i propose we take up esammer on his offer
>>> apurtell    so branch 0.92 asap, get trunk modular and working against multiple
platform targets
>>> apurtell    especially if we're going to see rpc changes coming from downstream
projects...
>>> apurtell    also what about supporting secure and nonsecure clients with the
same deployment?
>>> apurtell    zookeeper does this
>>> apurtell    so that is selectable rpc engine per connection, with a negotiation
>>> apurtell    we don't have or want to be crazy about it but a rolling upgrade
should be possible if for example we are taking in a new rpc from fb (?) or cloudera (avro
based?)
>>> apurtell    also looks like hlog modules for 0.20 vs 0.22 and successors
>>> apurtell    i think over time we can roadmap the rpc engines, if we have multiple,
by deprecation
>>> apurtell    now that we're on the edge of supporting both 0.20 and 0.22, and
secure vs nonsecure, let's get it as manageable as possible right away
>>>
>>> St^Ack_        apurtell: +1
>>>
>>> apurtell    also i think there is some interest in async rpc engine
>>>
>>> St^Ack_        we should stick this up on dev i'd say
>>>
>>> Best regards,
>>>
>>>     - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting
>>> back. - Piet Hein (via Tom White)
>>>
>>
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message