hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茅旭峰 <m9s...@gmail.com>
Subject HMaster startup is very slow, and always run into out-of-memory issue
Date Wed, 09 Mar 2011 10:05:33 GMT
Hi,

I'm setting a 4-region-nodes hbase clusters, with the master running outside
the data clusters.
It works 'well'. The problem is after some sort of stress tests, say
launching 20 threads, putting
data with restful api, each block of 4mb size, the hmaster always shut down
due to zookeeper
session timeout. After the timeout, I tried to restart the hmaster, then I
saw lots of

====
2011-03-09 17:56:16,032 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=107 entries
from
hdfs://cloud135:9000/hbase/.logs/cloud136,60020,1299572235779/cloud136%3A60020.1299636326073
2011-03-09 17:56:16,032 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 134 of
219:
hdfs://cloud135:9000/hbase/.logs/cloud136,60020,1299572235779/cloud136%3A60020.1299636333037,
length=96233748
2011-03-09 17:56:16,032 INFO org.apache.hadoop.hbase.util.FSUtils:
Recovering file
hdfs://cloud135:9000/hbase/.logs/cloud136,60020,1299572235779/cloud136%3A60020.1299636333037
2011-03-09 17:56:18,609 INFO org.apache.hadoop.hbase.util.FSUtils: Finished
lease recover attempt for
hdfs://cloud135:9000/hbase/.logs/cloud136,60020,1299572235779/cloud136%3A60020.1299636333037
====

in the log file, this recovery would take pretty long time, and it would
eventually lead to java heap out of memory,
even I set HBASE_HEAPSIZE to 4000 in conf/hbase-site.xml.

My question is the recovery takes so long time, and in the meanwhile, I
could not do anything, say bin/hbase shell, and list.
Is this normal? or some sort of configuration issue? If it's normal, is
there any guide line or document address this performance
issue? Thanks a lot for your reply.

I'm using hbase-0.90.1-CDH3B4.

Best regards,

Mao Xu-Feng

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message