Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates
 209.85.212.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADY20s4oX+WhTGptFouwLAEMVhJhC_nO4Tfr_dkwG5kAjsj+2g@mail.gmail.com>
References: 
 <CAKYwJ9x4U=SPGdSKAHKnOUwZ0upanQ7yjzC5QqMRuwAB9xFdAg@mail.gmail.com>
	<CADY20s4oX+WhTGptFouwLAEMVhJhC_nO4Tfr_dkwG5kAjsj+2g@mail.gmail.com>
Date: Sat, 30 Jun 2012 01:27:03 -0700
Message-ID: 
 <CAK+1EbhC6zFAOZiAB13P65W7z20KyOmzB3UxKFp70EyHrPOk0A@mail.gmail.com>
Subject: Re: HBASE-2182
From: Ryan Rawson <ryanobjc@gmail.com>
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Jun 29, 2012 at 5:04 PM, Todd Lipcon <todd@cloudera.com> wrote:
> A few inline notes below:
>
> On Fri, Jun 29, 2012 at 4:42 PM, Elliott Clark <eclark@stumbleupon.com>wr=
ote:
>
>> I just posted a pretty early skeleton(
>> https://issues.apache.org/jira/browse/HBASE-2182) on what I think a nett=
y
>> based hbase client/server could look like.
>>
>> Pros:
>>
>> =A0 - Faster
>> =A0 =A0 =A0- Giraph got a 3x perf improvement by droppping hadoop rpc
>>
>
> Whats the reference for this? The 3x perf I heard about from Giraph was
> from switching to using LMAX's Disruptor instead of queues, internally. W=
e
> could do the same, but I'm not certain the model works well for our use
> cases where the RPC processing can end up blocked on disk access, etc.
>
>
>> =A0 =A0 =A0- Asynhbase trounces our client when JD benchmarked them
>>
>
> I'm still convinced that the majority of this has to do with the way our
> batching happens to the server, not async vs sync. (in the current sync
> client, once we fill up the buffer, we "flush" from the same thread, and
> block the flush until all buffered edits have made it, vs doing it in the
> background). We could fix this without going to a fully async model.

I also agree here, if you do the apriori code analysis, it becomes
obvious that the issue is that slower regionservers can hold up entire
batches even if 90%+ of the Puts were already acked...

And don't forget that we used to issue Puts to regionservers SERIALLY
until we do the current parallelism code... (not that the code is
great, but it was relatively easy to fix at the time).


>
>
>> =A0 - Could encourage things to be a little more modular if everything i=
sn't
>> =A0 hanging directly off of HRegionServer
>>
> Sure, but not sure I see why this is Netty vs not-Netty
>
>
>> =A0 - Netty is better about thread usage than hadoop rpc server.
>>
> Can you explain further?
>
>
>> =A0 - Pretty easy to define an rpc protocol after all of the work on
>> =A0 protobuf (Thanks everyone)
>> =A0 - Decoupling the rpc server library from the hadoop library could al=
low
>> =A0 us to rev the server code easier.
>> =A0 - The filter model is very easy to work with.
>> =A0 =A0 =A0- Security can be just a single filter.
>> =A0 =A0 =A0- Logging can ba another
>> =A0 =A0 =A0- Stats can be another.
>>
>> Cons:
>>
>> =A0 - Netty and non apache rpc server's don't play well togther. =A0They=
 might
>> =A0 be able to but I haven't gotten there yet.
>>
> What do you mean "non apache rpc servers"?
>
>
>> =A0 - Complexity
>> =A0 =A0 =A0- Two different servers in the src
>> =A0 =A0 =A0- Confusing users who don't know which to pick
>> =A0 - Non-blocking could make the client a harder to write.
>>
>>
>> I'm really just trying to gauge what people think of the direction and i=
f
>> it's still something that is wanted. =A0The code is a loooooong way from=
 even
>> being a tech demo, and I'm not a netty expert, so suggestions would be
>> welcomed.
>>
>> Thoughts ? Are people interested in this? Should I push this to my githu=
b
>> so other can help ?
>>
>
> IMO, I'd want to see a noticeable perf difference from the change -
> unfortunately it would take a fair amount of work to get to the point whe=
re
> you could benchmark it. But if you're willing to spend the time to get to
> that point, seems worth investigating.
>
> --
> Todd Lipcon
> Software Engineer, Cloudera