Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 74.125.83.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=w516AuQgkq6qLPrjiJ3bO9klrOyFpRNGyxlxMLdrIN4MX2iKFe6OCegKqOUVvH2OHe
         M4cu+d0K1uRxxH1SN2G6Cm+jGM57muWOCiII4M+vPn36846tiC7yoxEvnJNRHUXVl0jk
         DC1DtmGFBKmipWW/+GvdsLCCm1+z7XdpS6MUM=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <727692.55730.qm@web33307.mail.mud.yahoo.com>
References: <727692.55730.qm@web33307.mail.mud.yahoo.com>
Date: Fri, 8 Apr 2011 14:20:08 -0700
Message-ID: <BANLkTinBw_9WnxqgRAGDVseV=7NYc3uvtA@mail.gmail.com>
Subject: Re: HTable.put hangs on bulk loading
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org, Ajay Govindarajan <agovindarajan@yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

That exception means you are running out of threads on that whole
machine. I wonder how you were able to get that... is hbase running on
that machine too? I'd love you see your configuration but what you
pasted is the hbase-default, which doesn't say anything since it's all
the default values.

The newer FC releases have an awful small setting for nproc If that's
your case you might want to bump that, google should tell you how on
your specific system.

J-D

On Fri, Apr 8, 2011 at 12:21 PM, Ajay Govindarajan
<agovindarajan@yahoo.com> wrote:
> I used to call HTable.close on each put. I commented it out and I get the=
 exception below (the program stops insertion at the exact same point i.e. =
15876 rows)
>
> Exception in thread "main" java.lang.OutOfMemoryError: unable to create n=
ew native thread
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.start0(Native Method)
> =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.start(Thread.java:640)
> =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor.ensureQu=
euedTaskHandled(ThreadPoolExecutor.java:760)
> =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor.execute(=
ThreadPoolExecutor.java:655)
> =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.AbstractExecutorService.sub=
mit(AbstractExecutorService.java:92)
> =A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.client.HConnectionManage=
r$TableServers.processBatchOfPuts(HConnectionManager.java:1439)
> =A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.client.HTable.flushCommi=
ts(HTable.java:664)
> =A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.client.HTable.doPut(HTab=
le.java:549)
> =A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.client.HTable.put(HTable=
.java:535)
>
> I am not sure why this happens because I commit after every put.
>
> Any help will be appreciated.
>
> thanks
> -ajay
>
>
>
>
>
> ________________________________
> From: Ajay Govindarajan <agovindarajan@yahoo.com>
> To: "user@hbase.apache.org" <user@hbase.apache.org>
> Sent: Thursday, April 7, 2011 5:35 PM
> Subject: Re: HTable.put hangs on bulk loading
>
> Thanks for pointing this out. I have uploaded the server config at:
> http://pastebin.com/U41QZGiq
>
> thanks
> -ajay
>
>
>
>
>
>
>
> ________________________________
> From: Jean-Daniel Cryans <jdcryans@apache.org>
> To: user@hbase.apache.org
> Sent: Thursday, April 7, 2011 10:29 AM
> Subject: Re: HTable.put hangs on bulk loading
>
> There's nothing of use in the pasted logs unfortunately, and the log
> didn't get attached to your mail (happens often). Consider putting on
> a web server or pastebin.
>
> Also I see you are on an older version, upgrading isn't going to fix
> your issue (which is probably related to your environment or
> configuration) but at least it's gonna be easier for us to support
> you.
>
> J-D
>
> On Wed, Apr 6, 2011 at 7:10 PM, ajay.gov <agovindarajan@yahoo.com> wrote:
>>
>> I am doing a load test for which I need to load a table with many rows. =
=A0I
>> have a small java program that has a for loop and calls HTable.put. =A0I=
 am
>> inserting a map of 2 items into a table that has one column family. The
>> limit of the for loop is currently 20000. However after 15876 rows the c=
all
>> to Put hangs. I am using autoFlush on the HTable. Any ideas why this may
>> happen?
>>
>> The table configuration:
>> DESCRIPTION =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0ENABLED
>> =A0{NAME =3D> 'TABLE2', FAMILIES =3D> [{NAME =3D> 'TABLE2_CF true
>> =A01', BLOOMFILTER =3D> 'NONE', REPLICATION_SCOPE =3D> '0'
>> =A0, COMPRESSION =3D> 'NONE', VERSIONS =3D> '3', TTL =3D> '2
>> =A0147483647', BLOCKSIZE =3D> '65536', IN_MEMORY =3D> 'fal
>> =A0se', BLOCKCACHE =3D> 'true'}]}
>>
>> The HBase config on the client is the one in the hbase-default.xml. Some
>> values:
>> hbase.client.write.buffer=3D2097152
>> hbase.client.pause=3D1000
>> hbase.client.retries.number=3D10
>>
>> If i use another client I am able to put items to the table. I am also a=
ble
>> to scan items from the table using the hbase shell.
>>
>> I have attached the server configuratio
>> I don't see anything in the region server or master logs. I have them he=
re.
>>
>> The master server log:
>> 2011-04-06 19:02:40,149 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server:
>> 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: <>}
>> 2011-04-06 19:02:40,152 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
>> 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: <>} comp=
lete
>> 2011-04-06 19:02:40,157 INFO org.apache.hadoop.hbase.master.ServerManage=
r: 1
>> region servers, 0 dead, average load 42.0
>> 2011-04-06 19:03:15,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {server:
>> 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: <>}
>> 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 40 row(s) of meta region {server:
>> 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: <>}
>> complete
>> 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:=
 All
>> 1 .META. region(s) scanned
>>
>>
>> The region server logs:
>> 2011-04-06 19:02:21,294 DEBUG org.apache.hadoop.hbase.regionserver.HRegi=
on:
>> Creating region TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>> 2011-04-06 19:02:21,295 INFO org.apache.hadoop.hbase.regionserver.HRegio=
n:
>> Onlined TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>> sequenceid=3D1
>>
>> --
>> View this message in context: http://old.nabble.com/HTable.put-hangs-on-=
bulk-loading-tp31338874p31338874.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>