hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: 0.90 latency performance, cdh3b4
Date Fri, 22 Apr 2011 06:09:20 GMT
Dmitriy,

Did I hear you say that you are instantiating a new Htable for each request?
 Or was that somebody else?

On Thu, Apr 21, 2011 at 11:04 PM, Stack <stack@duboce.net> wrote:

> On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> wrote:
> > Anyway. For a million requests shot at a region server at various
> > speeds between 300 and 500 qps the picture is not pretty. RPC metrics
> > are arctually good -- no more than 1ms average per next() and 0 per
> > get(). So region server is lightning fast.
> >
>
> This is 3-500 queries per second of 40 rows each?
>
> > What doesn't seem so fast is RPC.
>
> OK.
>
> > As i reported before, i was getting
> > 25ms TTLB under the circumstances. In this case all the traffic to the
> > node goes thru same client (but in reality of course the node's
> > portion per client should be much less). All that traffic is using
> > single regionserver node rpc queue as HConnection would not open more
> > than one socket to same region.
>
> You saw "HBASE-2939  Allow Client-Side Connection Pooling"?  Would that
> help?
>
>
> > And tcp doesn't seem to perform very
> > well for some reason in this scenario.
> >
>
> What do you see here D?
>
>
> > The next thing i did was to enable tcp_nodelay on both client and
> > server. That got us down even more to 13ms average.
> >
>
> Thats a big difference.
>
>
> > However, it is still about two times slower if i run all processes at
> > the same machine (i get around 6-7ms average TTLBs for the same type
> > of scan).
> >
> > Ping time for about same packet size between hosts involved seems to
> > revolve around 1ms. Where another 5ms average time are getting lost is
> > still a mystery. But oh well i guess it is as good as it gets.
> > In real life hbase applications traffic would be much more uniformly
> > distributed among regions and this would be much less of an issue
> > perhaps.
> >
> > I also suspect that using udp for short scans and gets might reduce
> > latency a bit as well.
> >
>
> Thank you Dmitriy for digging in.  Good stuff.
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message