hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <la...@apache.org>
Subject Re: Fwd: High CPU utilization in few Region servers during read
Date Fri, 25 Oct 2013 05:16:40 GMT
No this is different.

All your data is in the memstore still.

The memstore is organized as a skip list, nobody has ever tested that with 72gb. 256mb, 512mb,
1gb, sure... 72gb... no way.
Same with a 96gb of java heap. Not with Oracle or OpenJDK and an application specifically
for such large heaps.
I would keep it under 30gb.

I think what you want is the following:
1. disable WAL writes (you don't care if you lose data)
2. lower your memstore size, so you'll see some flushes and eventually compactions.

3. Don't give the JVM more the 30g or so
4. Flush your memstores to disk. They'll end up in the block cache that way

Currently we can't fill the block cache without flushing to disk.

Maybe HBase is not the right solution. If you need a large ephemeral in-memory store, maybe
look at memcache?

-- Lars

 From: Vinay Kashyap <vinu.kash@gmail.com>
To: user@hbase.apache.org 
Sent: Wednesday, October 23, 2013 5:57 PM
Subject: Fwd: High CPU utilization in few Region servers during read

>From the thread dump looks like so many threads are stuck at





Is this similar to HBASE-9428 issue.??

Waiting for some help regarding this.. :)

Thanks and regards
Vinay S Kashyap

---------- Forwarded message ----------
From: Vinay Kashyap <vinu.kash@gmail.com>
Date: Tue, Oct 22, 2013 at 8:47 PM
Subject: High CPU utilization in few Region servers during read
To: user@hbase.apache.org


I am running HBase 0.94.6 (cdh-4.4.0) with 25 region servers.
I am testing a scenario to read and write only from/to RAM.

I have the following settings
Table precreated with 25 regions.
HFile size - 48 GB
MemStore size - 72 GB
Heap size - 96 GB

These settings are to avoid any flushes to the disk. Data need not be

I am able to achieve a load throughput of 75K ops per region server.
While reading 23 region servers are serving requests with throughput of 55K
ops, but randomly 2 of the region servers always end up serving few 100 ops.

In these 2 region servers the CPU usage is very high and close to 100%
continuously bringing down the overall throughput. I did not observe any
long GC pauses in this time.

I also tried applying the patch for HBASE-9428 issue, but still faced the
same problem.
Thread dump for the affected region server is at

Any hints on how to solve this.?

Thanks and regards
Vinay S Kashyap
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message