Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates
 209.85.214.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BANLkTikdxQxGEF1h+p-cbKjRezkpnfV5Jw@mail.gmail.com>
References: <479265.45500.qm@web65505.mail.ac4.yahoo.com>
 <167353.65612.qm@web65509.mail.ac4.yahoo.com>
 <BANLkTikdxQxGEF1h+p-cbKjRezkpnfV5Jw@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 27 May 2011 13:30:09 -0700
Message-ID: <BANLkTinYMV7pxdYCCT=usoqbgWv8wKYi3A@mail.gmail.com>
Subject: Re: modular build and pluggable rpc
To: dev@hbase.apache.org
Cc: apurtell@apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Agreed - I'm all for Thrift.

Though, I actually, contrary to Ryan, think that the existing HBaseRPC
handler/client code is pretty good -- better than the equivalents from
Thrift Java.

We could start by using Thrift serialization on our existing transport
-- then maybe work towards contributing it upstream to the Thrift
project. HDFS folks are potentially interested in doing that as well.

-Todd

On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> I'm -1 on avro as a RPC format. =A0Thrift is the way to go, any of the
> advantages of smaller serialization of avro is lost by the sheer
> complexity of avro and therefore the potential bugs.
>
> I understand the desire to have a pluggable RPC engine, but it feels
> like the better approach would be to adopt a unified RPC and just be
> done with it. =A0I had a look at the HsHa mechanism in thrift and it is
> very good, it in fact matches our 'handler' approach - async
> recieving/sending of data, but single threaded for processing a
> message.
>
> -ryan
>
> On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <apurtell@apache.org> wro=
te:
>> Also needing, perhaps later, consideration:
>>
>> - HDFS-347 or not
>>
>> =A0- Lucene embedding for hbase-search, though as a coprocessor this is =
already pretty much handled if we have platform support (therefore a platfo=
rm module) for a HDFS that can do local read shortcutting and block placeme=
nt requests
>>
>> - HFile v1 versus v2
>>
>> Making decoupled development at several downstream sites manageable, wit=
h a home upstream for all the work, while simultaneously providing clean mi=
gration paths for users, basically.
>>
>> --- On Fri, 5/27/11, Andrew Purtell <apurtell@apache.org> wrote:
>>
>>> From: Andrew Purtell <apurtell@apache.org>
>>> Subject: modular build and pluggable rpc
>>> To: dev@hbase.apache.org
>>> Date: Friday, May 27, 2011, 12:49 PM
>>> From IRC:
>>>
>>> apurtell=A0=A0=A0 i propose we take the build modular as early as possi=
ble to deal with multiple platform targets
>>> apurtell=A0=A0=A0 secure vs nonsecure
>>> apurtell=A0=A0=A0 0.20 vs 0.22 vs trunk
>>> apurtell=A0=A0=A0 i understand the maintenence issues with multiple rpc=
 engines, for example, but a lot of reflection twistiness is going to be wo=
rse
>>> apurtell=A0=A0=A0 i propose we take up esammer on his offer
>>> apurtell=A0=A0=A0 so branch 0.92 asap, get trunk modular and working ag=
ainst multiple platform targets
>>> apurtell=A0=A0=A0 especially if we're going to see rpc changes coming f=
rom downstream projects...
>>> apurtell=A0=A0=A0 also what about supporting secure and nonsecure clien=
ts with the same deployment?
>>> apurtell=A0=A0=A0 zookeeper does this
>>> apurtell=A0=A0=A0 so that is selectable rpc engine per connection, with=
 a negotiation
>>> apurtell=A0=A0=A0 we don't have or want to be crazy about it but a roll=
ing upgrade should be possible if for example we are taking in a new rpc fr=
om fb (?) or cloudera (avro based?)
>>> apurtell=A0=A0=A0 also looks like hlog modules for 0.20 vs 0.22 and suc=
cessors
>>> apurtell=A0=A0=A0 i think over time we can roadmap the rpc engines, if =
we have multiple, by deprecation
>>> apurtell=A0=A0=A0 now that we're on the edge of supporting both 0.20 an=
d 0.22, and secure vs nonsecure, let's get it as manageable as possible rig=
ht away
>>>
>>> St^Ack_=A0=A0=A0 =A0=A0=A0 apurtell: +1
>>>
>>> apurtell=A0=A0=A0 also i think there is some interest in async rpc engi=
ne
>>>
>>> St^Ack_=A0=A0=A0 =A0=A0=A0 we should stick this up on dev i'd say
>>>
>>> Best regards,
>>>
>>> =A0 =A0 - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting
>>> back. - Piet Hein (via Tom White)
>>>
>>
>


--=20
Todd Lipcon
Software Engineer, Cloudera