Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.26 as permitted sender)
Message-ID: <31485864.post@talk.nabble.com>
Date: Wed, 27 Apr 2011 02:30:10 -0700 (PDT)
From: Stan Barton <bartx007@gmail.com>
To: hbase-user@hadoop.apache.org
Subject: Re: HTable.put hangs on bulk loading
In-Reply-To: <31480444.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <31338874.post@talk.nabble.com> <31345580.post@talk.nabble.com>
 <31477194.post@talk.nabble.com> <31480444.post@talk.nabble.com>


Hi,

what means increase? I checked on the client machines and the nproc limit is
around 26k, that seems to be as sufficient. The same limit applies on the db
machines...

Stan


ajay.gov wrote:
> 
> Hi,
> 
> I posted the same message on the user@hbase.apache.org mailing list and
> Jean-Daniel Cryans suggested i increase the nproc limit on the client
> machines. I did it and it fixed the problem.
> 
> -ajay
> 
> 
> 
> Stan Barton wrote:
>> 
>> I am running into similar problem with HBase 0.90.2. My setting is 6 RSs
>> one master server, 3 ZK servers and about 20 clients (on different
>> servers) feeding the database with inserts. After few hours and around
>> 2.5 millions rows inserted the process simply hangs with no what so ever
>> error indication (neither by ZK, master, regionservers nor the clients).
>> All the inserting clients stop virtually at the same moment, but HBase it
>> self is not down and can be queried. 
>> 
>> In fact, the clients not even crash on time out exception for its
>> connection. I have ran into such problem always when I attempted to run
>> such importing progress with the new HBase versions (even 0.90.1). Can
>> anybody address this problem? Does other have similar problems?
>> 
>> I can provide further info about the configuration if needed.
>> 
>> Stan Barton
>> 
>> 
>> 
>> 
>> ajay.gov wrote:
>>> 
>>> Sorry, my server config was not attached. Its here:
>>> http://pastebin.com/U41QZGiq
>>> 
>>> thanks
>>> -ajay
>>> 
>>> 
>>> 
>>> ajay.gov wrote:
>>>> 
>>>> I am doing a load test for which I need to load a table with many rows. 
>>>> I have a small java program that has a for loop and calls HTable.put. 
>>>> I am inserting a map of 2 items into a table that has one column
>>>> family. The limit of the for loop is currently 20000. However after
>>>> 15876 rows the call to Put hangs. I am using autoFlush on the HTable.
>>>> Any ideas why this may happen? 
>>>> 
>>>> The table configuration:
>>>> DESCRIPTION                                          ENABLED                    
>>>>  {NAME => 'TABLE2', FAMILIES => [{NAME => 'TABLE2_CF true                       
>>>>  1', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0'                            
>>>>  , COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2                            
>>>>  147483647', BLOCKSIZE => '65536', IN_MEMORY => 'fal                            
>>>>  se', BLOCKCACHE => 'true'}]}
>>>> 
>>>> The HBase config on the client is the one in the hbase-default.xml.
>>>> Some values:
>>>> hbase.client.write.buffer=2097152
>>>> hbase.client.pause=1000
>>>> hbase.client.retries.number=10
>>>> 
>>>> If i use another client I am able to put items to the table. I am also
>>>> able to scan items from the table using the hbase shell.
>>>> 
>>>> I have attached the server configuratio
>>>> I don't see anything in the region server or master logs. I have them
>>>> here.
>>>> 
>>>> The master server log:
>>>> 2011-04-06 19:02:40,149 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
>>>> scanning meta region {server: 184.106.69.238:60020, regionname:
>>>> -ROOT-,,0.70236052, startKey: <>}
>>>> 2011-04-06 19:02:40,152 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
>>>> scan of 1 row(s) of meta region {server: 184.106.69.238:60020,
>>>> regionname: -ROOT-,,0.70236052, startKey: <>} complete
>>>> 2011-04-06 19:02:40,157 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead,
>>>> average load 42.0
>>>> 2011-04-06 19:03:15,252 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
>>>> scanning meta region {server: 184.106.69.238:60020, regionname:
>>>> .META.,,1.1028785192, startKey: <>}
>>>> 2011-04-06 19:03:15,265 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
>>>> scan of 40 row(s) of meta region {server: 184.106.69.238:60020,
>>>> regionname: .META.,,1.1028785192, startKey: <>} complete
>>>> 2011-04-06 19:03:15,266 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
>>>> scanned
>>>> 
>>>> 
>>>> The region server logs:
>>>> 2011-04-06 19:02:21,294 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>>>> 2011-04-06 19:02:21,295 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>>>> sequenceid=1
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31485864.html
Sent from the HBase User mailing list archive at Nabble.com.