Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
From: <Michael.Grundvig@high5games.com>
To: <user@hbase.apache.org>
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine
Thread-Topic: HBase Client Performance Bottleneck in a Single Virtual Machine
Thread-Index: 
 Ac7ZD2fWarnxlPypREuBERe3xhwp2QAf0H1NAAAZydAAF+MFAAABW/GAAAGXGoAADrDm8AAMvR+AAAjlQOA=
Date: Tue, 5 Nov 2013 16:35:22 +0000
Message-ID: <1B3B6E390CD2604483EADB54945CC15212EBFA2A@HIGH5-EXCH1>
References: <1B3B6E390CD2604483EADB54945CC15212EBD5BD@HIGH5-EXCH1>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42AFD@exchange2007.carrieriq.com>
 <1B3B6E390CD2604483EADB54945CC15212EBE0E6@HIGH5-EXCH1>
 <1383614210.70902.YahooMailNeo@web140601.mail.bf1.yahoo.com>,<1383616545.37906.YahooMailNeo@web140604.mail.bf1.yahoo.com>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42B02@exchange2007.carrieriq.com>
 <1B3B6E390CD2604483EADB54945CC15212EBF7A8@HIGH5-EXCH1>
 <1383666401.99243.YahooMailNeo@web140601.mail.bf1.yahoo.com>
In-Reply-To: <1383666401.99243.YahooMailNeo@web140601.mail.bf1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I'm finding this configuration process very confusing. This is what we are =
doing in Java code:

        Configuration configuration =3D HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", props.getProperty(HBASE=
_NODES));
        configuration.set("zookeeper.znode.parent", props.getProperty(HBASE=
_PARAMS));
        configuration.setBoolean(" hbase.ipc.client.tcpnodelay", true);
        configuration.setInt("hbase.client.ipc.pool.size", 10);

Adding those last two lines has made no improvement. Likewise, updating hba=
se-site.xml on the cluster to include these settings has made no improvemen=
t. The problem is that I have zero confidence it's actually taking the sett=
ings in the first place.=20

-Mike


-----Original Message-----
From: lars hofhansl [mailto:larsh@apache.org]=20
Sent: Tuesday, November 05, 2013 9:47 AM
To: user@hbase.apache.org
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machin=
e

Hi Mike,

hbase-site.xml has both client and server side settings.

In your client, where do you set the ZK quorum of the cluster? At that same=
 spot you'd set these options.

Assuming you're running the clients with classes from a normal HBase instal=
l, you would add these setting to the hbase-site.xml there.

-- Lars


________________________________
 From: "Michael.Grundvig@high5games.com" <Michael.Grundvig@high5games.com>
To: user@hbase.apache.org=20
Sent: Tuesday, November 5, 2013 6:43 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machin=
e
=20

Thanks for the input, we'll do some more testing on this today. Where do th=
ese settings get made? In the configuration used to create the pool or some=
where else? It appears hbase-site.xml "on the client" but we have nothing l=
ike that so I think I'm misunderstanding something. Thanks!

-Mike


-----Original Message-----
From: Vladimir Rodionov [mailto:vrodionov@carrieriq.com]=20
Sent: Monday, November 04, 2013 8:41 PM
To: user@hbase.apache.org; lars hofhansl
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machin=
e

One more: "hbase.ipc.client.tcpnodelay" set to true. It is worth trying as =
well.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: lars hofhansl [larsh@apache.org]
Sent: Monday, November 04, 2013 5:55 PM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machin=
e

Here're one more thing to try. By default each HConnection will use a singl=
e TCP connection to multiplex traffic to each region server.

Try setting hbase.client.ipc.pool.size on the client to something > 1.

-- Lars


________________________________
From: lars hofhansl <larsh@apache.org>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Sent: Monday, November 4, 2013 5:16 PM
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machin=
e


No. This is terrible.
If you can, please send a jstack and do some profiling. Is there an easy wa=
y to reproduce this with just a single RegionServer?
If so, I'd offer to do some profiling.

Thanks.


-- Lars


________________________________

From: "Michael.Grundvig@high5games.com" <Michael.Grundvig@high5games.com>
To: user@hbase.apache.org
Sent: Monday, November 4, 2013 11:00 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machin=
e


Not yet, this is just a load test client. It literally does nothing but cre=
ate threads to talk to HBase and run 4 different calls. Nothing else is don=
e in the app at all.

To eliminate even more of our code from the loop, we just tried removing ou=
r connection pool entirely and just using a single connection per thread - =
no improvement. Then we tried creating the HTableInterface (all calls are a=
gainst the same table) at the time of connection creation. The means thread=
 to connection to table interface were all at 1 to 1 and not being passed a=
round. No performance improvement.

Long story short, running a single thread it's fast. Start multithreading, =
it starts slowing down. CPU usage, memory usage, etc. are all negligible. T=
he performance isn't terrible - it's probably good enough for the vast majo=
rity of users, but it's not good enough for our app. With one thread, it mi=
ght take 5 milliseconds. With 10 threads all spinning more quickly (40 mill=
iseconds delay), the call time increases to 15-30 milliseconds. The problem=
 is that at our throughput rates, that's a serious concern.

We are going to fire up a profiler next to see what we can find.

-Mike


-----Original Message-----
From: Vladimir Rodionov [mailto:vrodionov@carrieriq.com]
Sent: Monday, November 04, 2013 12:50 PM
To: user@hbase.apache.org
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machin=
e

Michael, have you tried jstack on your client application?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Michael.Grundvig@high5games.com [Michael.Grundvig@high5games.com]
Sent: Sunday, November 03, 2013 7:46 PM
To: user@hbase.apache.org
Subject: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi all; I posted this as a question on StackOverflow as well but realized I=
 should have gone straight ot the horses-mouth with my question. Sorry for =
the double post!

We are running a series of HBase tests to see if we can migrate one of our =
existing datasets from a RDBMS to HBase. We are running 15 nodes with 5 zoo=
keepers and HBase 0.94.12 for this test.

We have a single table with three column families and a key that is distrib=
uting very well across the cluster. All of our queries are running a direct=
 look-up; no searching or scanning. Since the HTablePool is now frowned upo=
n, we are using the Apache commons pool and a simple connection factory to =
create a pool of connections and use them in our threads. Each thread creat=
es an HTableInstance as needed and closes it when done. There are no leaks =
we can identify.

If we run a single thread and just do lots of random calls sequentially, th=
e performance is quite good. Everything works great until we start trying t=
o scale the performance. As we add more threads and try and get more work d=
one in a single VM, we start seeing performance degrade quickly. The client=
 code is simply attempting to run either one of several gets or a single pu=
t at a given frequency. It then waits until the next time to run, we use th=
is to simulate the workload from external clients. With a single thread, we=
 will see call times in the 2-3 milliseconds which is acceptable.

As we add more threads, this call time starts increasing quickly. What gets=
 strange is if we add more VMs, the times hold steady across them all so cl=
early it's a bottleneck in the running instance and not the cluster. We can=
 get a huge amount of processing happening across the cluster very easily -=
 it just has to use a lot of VMs on the client side to do it. We know the c=
ontention isn't in the connection pool as we see the problem even when we h=
ave more connections than threads. Unfortunately, the times are spiraling o=
ut of control very quickly. We need it to support at least 128 threads in p=
ractice, but most important I want to support 500 updates/sec and 250 gets/=
sec. In theory, this should be a piece of cake for the cluster as we can do=
 FAR more work than that with a few VMs, but we don't even get close to thi=
s with a single VM.

So my question: how do people building high-performance apps with HBase get=
 around this? What approach are others using for connection pooling in a mu=
lti-threaded environment? There seems to be a surprisingly little amount of=
 info about this on the web considering the popularity. Is there some clien=
t setting we need to use that makes it perform better in a threaded environ=
ment? We are going to try to cache HTable instances next but that's a total=
 guess. There are solutions to offloading work to other VMs but we really w=
ant to avoid this as clearly the cluster can handle the load and it will dr=
amatically decrease the application performance in critical areas.

Any help is greatly appreciated! Thanks!
-Mike

Confidentiality Notice:=A0 The information contained in this message, inclu=
ding any attachments hereto, may be confidential and is intended to be read=
 only by the individual or entity to whom this message is addressed. If the=
 reader of this message is not the intended recipient or an agent or design=
ee of the intended recipient, please note that any review, use, disclosure =
or distribution of this message or its attachments, in any form, is strictl=
y prohibited.=A0 If you have received this message in error, please immedia=
tely notify the sender and/or Notifications@carrieriq.com and delete or des=
troy any copy of this message and its attachments.