hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Pallas <joseph.pal...@oracle.com>
Subject Re: HBase multi-threaded client consumes a lot of threads
Date Thu, 21 Jul 2011 17:17:25 GMT

On Jul 21, 2011, at 6:08 AM, Steinmaurer Thomas wrote:

> The client can be configured on how many concurrent clients (one thread
> per client), number of records per client etc ... The thing is, even
> with only one client (thread), the java process consumes a lot more
> threads, which is getting beyond 1000 threads when simulating a load
> with 100 clients with this test application.

Are you using HTablePool?  If not, you may get excess threads if you create many HTable instances
and don't call close on them.  That was the cause of my thread explosion.  Each HTable instance
has a ThreadPoolExecutor with a 60-second timeout, so if you don't call close after your work
is done, an extra thread will be left idle until the timeout expires.  (And if you're not
on 0.90.3 with HBASE-3712 fixed, even calling close won't help.)

Using HTablePool solves the problem by limiting the number of HTable instances that are created
and recycling them.

joe


Mime
View raw message