hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhaval Shah <prince_mithi...@yahoo.co.in>
Subject Re: RegionServer crash without any errors (compaction?)
Date Thu, 07 Nov 2013 16:21:15 GMT
Operation too slow is generally in the .log file while the GC logs (if you enabled GC logging)
is in the .out file. You have a very small heap for a 1GB HFIle size. You are probably running
your region server out of memory. Try increasing the heap size and see if that helps

 From: John <johnnyenglish739@gmail.com>
To: user@hbase.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in> 
Sent: Thursday, 7 November 2013 11:09 AM
Subject: Re: RegionServer crash without any errors (compaction?)

there are no really other logs before. There are a "operationTooSlow" message before, but
that log is ~50 mins bofre the other: http://pastebin.com/EAAubqGB

2013/11/7 John <johnnyenglish739@gmail.com>

>thanks for your fast answer. If I take a look at the cloudera manager at this time the
%-time of using the GC increase at this time, so I think you are right. The max heap size
is 1GB for this node. The hbase.hregion.max.filesize is also 1GB. 
>2013/11/7 Dhaval Shah <prince_mithibai@yahoo.co.in>
>Did you look at your GC logs? Probably the compaction process is running your region server
out of memory. Can you provide more details on your setup? Max heap size? Max Region HFile
>> From: John <johnnyenglish739@gmail.com>
>>To: user@hbase.apache.org
>>Sent: Thursday, 7 November 2013 10:51 AM
>>Subject: RegionServer crash without any errors (compaction?)
>>I have a cluster with 7 regionserver. Some of them are crashing from time
>>to time wihtout any error message in the hbase log. If I take a look at the
>>log at the time I found this:
>>2013-11-07 15:29:02,511 INFO org.apache.hadoop.hbase.regionserver.Store:
>>Starting compaction of 2 file(s) in 1 of P_SO,<
>>2013-11-07 15:29:10,471 INFO
>>org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter
>>type for hdfs://
>>2013-11-07 15:31:05,944 INFO org.apache.hadoop.hbase.util.VersionInfo:
>>HBase 0.94.6-cdh4.4.0
>>.... restart
>>At this time 2 of the 7 RS crashed, both has this compaction message before
>>they crashed. I don't know exactly what compaction is, but it seems that
>>this compaction has to do with the crash. What can I do to avoid this
>>best regards
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message