Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates
 74.125.83.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=IT0e3W94i21nKlBMnDlpXf3QtlZiaXGv320zFwDzKpKjvi83i2z5vVBZpnaMcdWx4z
         sE9o7fdmireqbHcq7buJjOdBS7Zc+M4ImBWpXAROLBiGtk5OFS8NtY1PZvACpqIvnsSp
         zua/52D4FJ/cGOPupmpV/+J3jF5/HOB2wRZyU=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <BANLkTinpr8_xdx1G9fzfFxdeFeA=2U3hMA@mail.gmail.com>
References: <BANLkTinpr8_xdx1G9fzfFxdeFeA=2U3hMA@mail.gmail.com>
Date: Thu, 23 Jun 2011 10:03:57 -0700
Message-ID: <BANLkTikfUS-yRznLcrRFDK8mmpHGbeLbYQ@mail.gmail.com>
Subject: Re: checkAndPut() failing with NotServingRegionException
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Getting RetriesExhaustedWithDetailsException due to NSRE means that it
took forever for a region server to close or split a region, what you
pasted from the region server talks a region closing but that also
happens during split.

I'd suggest digging more in those region server logs using this guide:
http://hbase.apache.org/book/trouble.html

Also make sure you review this http://hbase.apache.org/book/performance.htm=
l

Finally giving a 1GB heap to HBase while inserting a lot of data is
like making a malnourished child work in a coal mine, it's not very
nice of you.

J-D

On Wed, Jun 22, 2011 at 11:06 PM, Sam Seigal <selekt86@yahoo.com> wrote:
> Hi,
>
> I am loading data into my HBase cluster and running into two issues -
>
> During my import, I received the following exception ->
>
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Fail=
ed
> 53484 actions: servers with issues: spock7001:60020,
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on.processBatch(HConnectionManager.java:1220)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementati=
on.processBatchOfPuts(HConnectionManager.java:1234)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java=
:675)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hbase.client.HTable.put(HTable.java:6=
60)
>
> May have cluster issues =3D> true
> Cause 0
>
> When I check the logs on the regions server, the last thrown exception is
> the following =3D>
>
> Thu Jun 23 05:16:18 2011 GMT regionserver 10460-0@spock7001:0 [DEBUG] (IP=
C
> Server handler 7 on 60020)
> { org.apache.hadoop.hbase.NotServingRegionException:
> hbaseTable,,1308805558566.5aefc6c2b9599f55f8b40351a61db03c. is closing
> Thu Jun 23 05:22:18 2011 GMT regionserver 10460-0@spock7001:0 [DEBUG]
> (regionserver60020.logRoller) org.apache.hadoop.conf.Configuration:
> java.io.IOException: config()
>
> On running status 'detailed' in the shell , I get =3D>
>
> 0 regionsInTransition
> 3 live servers
> =A0 spock7001:60020 1308805454136
> =A0 =A0 =A0 =A0requests=3D0, regions=3D0, usedHeap=3D470, maxHeap=3D910
> =A0 =A0spock6002:60020 1308805434201
> =A0 =A0 =A0 =A0requests=3D0, regions=3D1, usedHeap=3D550, maxHeap=3D910
> =A0 =A0 =A0 =A0hbaseTable,,1308805558566.5aefc6c2b9599f55f8b40351a61db03c=
.
> =A0 =A0 =A0 =A0 =A0 =A0stores=3D1, storefiles=3D2, storefileSizeMB=3D383,=
 memstoreSizeMB=3D0,
> storefileIndexSizeMB=3D1
> =A0 =A0spock6001:60020 1308805268507
> =A0 =A0 =A0 =A0requests=3D0, regions=3D2, usedHeap=3D90, maxHeap=3D910
> =A0 =A0 =A0 =A0-ROOT-,,0
> =A0 =A0 =A0 =A0 =A0 =A0stores=3D1, storefiles=3D1, storefileSizeMB=3D0, m=
emstoreSizeMB=3D0,
> storefileIndexSizeMB=3D0
> =A0 =A0 =A0 =A0.META.,,1
> =A0 =A0 =A0 =A0 =A0 =A0stores=3D1, storefiles=3D0, storefileSizeMB=3D0, m=
emstoreSizeMB=3D0,
> storefileIndexSizeMB=3D0
> 0 dead servers
>
>
> I am issuing a checkAndPut() to insert records into HBase. Is this a bug =
?
>
> Secondly, I have followed the instructions in the HBase book to increase
> write throughput. I have the following settings for my hbase table:
>
> config =3D HBaseConfiguration.create();
> table =3D new HTable (config, "hbaseTable");
> table.setAutoFlush(false);
> table.setWriteBufferSize(104857600);
>
> However, according to my logs, each checkAndPut() call takes on an averag=
e
> of 5 milliseconds. Is this unavoidable overhead due to locking ?
>
> All of my HBase daemons are running with -Xmx1g of heapsize.
>
> Any help is appreciated.
>
> Thank you,
>
> Sam
>