hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: checkAndPut() failing with NotServingRegionException
Date Thu, 23 Jun 2011 17:03:57 GMT
Getting RetriesExhaustedWithDetailsException due to NSRE means that it
took forever for a region server to close or split a region, what you
pasted from the region server talks a region closing but that also
happens during split.

I'd suggest digging more in those region server logs using this guide:
http://hbase.apache.org/book/trouble.html

Also make sure you review this http://hbase.apache.org/book/performance.html

Finally giving a 1GB heap to HBase while inserting a lot of data is
like making a malnourished child work in a coal mine, it's not very
nice of you.

J-D

On Wed, Jun 22, 2011 at 11:06 PM, Sam Seigal <selekt86@yahoo.com> wrote:
> Hi,
>
> I am loading data into my HBase cluster and running into two issues -
>
> During my import, I received the following exception ->
>
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
> 53484 actions: servers with issues: spock7001:60020,
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1220)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
>        at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
>        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
>
> May have cluster issues => true
> Cause 0
>
> When I check the logs on the regions server, the last thrown exception is
> the following =>
>
> Thu Jun 23 05:16:18 2011 GMT regionserver 10460-0@spock7001:0 [DEBUG] (IPC
> Server handler 7 on 60020)
> { org.apache.hadoop.hbase.NotServingRegionException:
> hbaseTable,,1308805558566.5aefc6c2b9599f55f8b40351a61db03c. is closing
> Thu Jun 23 05:22:18 2011 GMT regionserver 10460-0@spock7001:0 [DEBUG]
> (regionserver60020.logRoller) org.apache.hadoop.conf.Configuration:
> java.io.IOException: config()
>
> On running status 'detailed' in the shell , I get =>
>
> 0 regionsInTransition
> 3 live servers
>   spock7001:60020 1308805454136
>        requests=0, regions=0, usedHeap=470, maxHeap=910
>    spock6002:60020 1308805434201
>        requests=0, regions=1, usedHeap=550, maxHeap=910
>        hbaseTable,,1308805558566.5aefc6c2b9599f55f8b40351a61db03c.
>            stores=1, storefiles=2, storefileSizeMB=383, memstoreSizeMB=0,
> storefileIndexSizeMB=1
>    spock6001:60020 1308805268507
>        requests=0, regions=2, usedHeap=90, maxHeap=910
>        -ROOT-,,0
>            stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0
>        .META.,,1
>            stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0
> 0 dead servers
>
>
> I am issuing a checkAndPut() to insert records into HBase. Is this a bug ?
>
> Secondly, I have followed the instructions in the HBase book to increase
> write throughput. I have the following settings for my hbase table:
>
> config = HBaseConfiguration.create();
> table = new HTable (config, "hbaseTable");
> table.setAutoFlush(false);
> table.setWriteBufferSize(104857600);
>
> However, according to my logs, each checkAndPut() call takes on an average
> of 5 milliseconds. Is this unavoidable overhead due to locking ?
>
> All of my HBase daemons are running with -Xmx1g of heapsize.
>
> Any help is appreciated.
>
> Thank you,
>
> Sam
>

Mime
View raw message