hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: 0.90 latency performance, cdh3b4
Date Fri, 22 Apr 2011 06:14:15 GMT
in this case i pool them as well, which doesn't seem to make any
difference (compared to when i just reuse them -- but i am not writing
but outside of the test i do so i do pool them using techniques
similar to those in HTablePool, CAS-based queues etc. )


On Thu, Apr 21, 2011 at 11:09 PM, Ted Dunning <tdunning@maprtech.com> wrote:
> Dmitriy,
>
> Did I hear you say that you are instantiating a new Htable for each request?
>  Or was that somebody else?
>
> On Thu, Apr 21, 2011 at 11:04 PM, Stack <stack@duboce.net> wrote:
>
>> On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
>> wrote:
>> > Anyway. For a million requests shot at a region server at various
>> > speeds between 300 and 500 qps the picture is not pretty. RPC metrics
>> > are arctually good -- no more than 1ms average per next() and 0 per
>> > get(). So region server is lightning fast.
>> >
>>
>> This is 3-500 queries per second of 40 rows each?
>>
>> > What doesn't seem so fast is RPC.
>>
>> OK.
>>
>> > As i reported before, i was getting
>> > 25ms TTLB under the circumstances. In this case all the traffic to the
>> > node goes thru same client (but in reality of course the node's
>> > portion per client should be much less). All that traffic is using
>> > single regionserver node rpc queue as HConnection would not open more
>> > than one socket to same region.
>>
>> You saw "HBASE-2939  Allow Client-Side Connection Pooling"?  Would that
>> help?
>>
>>
>> > And tcp doesn't seem to perform very
>> > well for some reason in this scenario.
>> >
>>
>> What do you see here D?
>>
>>
>> > The next thing i did was to enable tcp_nodelay on both client and
>> > server. That got us down even more to 13ms average.
>> >
>>
>> Thats a big difference.
>>
>>
>> > However, it is still about two times slower if i run all processes at
>> > the same machine (i get around 6-7ms average TTLBs for the same type
>> > of scan).
>> >
>> > Ping time for about same packet size between hosts involved seems to
>> > revolve around 1ms. Where another 5ms average time are getting lost is
>> > still a mystery. But oh well i guess it is as good as it gets.
>> > In real life hbase applications traffic would be much more uniformly
>> > distributed among regions and this would be much less of an issue
>> > perhaps.
>> >
>> > I also suspect that using udp for short scans and gets might reduce
>> > latency a bit as well.
>> >
>>
>> Thank you Dmitriy for digging in.  Good stuff.
>> St.Ack
>>
>

Mime
View raw message