hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: HTable.put hangs on bulk loading
Date Tue, 26 Apr 2011 18:36:03 GMT
Would it make sense to do some kind of sanity check on these various configuration parameters
when a 
region or master server starts?  It seems like there is a lot of them and when they aren't
right, it 
can cause big problems.  Just have it check the configuration parameters and output a warning
in the 
log.  E.g. log.warn("Warning, OS setting may be too low: ulimit should be at least X.  Considering

changing it.");

Also if that were to show up in the Master.jsp it would be even better.

~Jeff

On 4/26/2011 11:17 AM, ajay.gov wrote:
> Hi,
>
> I posted the same message on the user@hbase.apache.org mailing list and
> Jean-Daniel Cryans suggested i increase the nproc limit on the client
> machines. I did it and it fixed the problem.
>
> -ajay
>
>
>
> Stan Barton wrote:
>> I am running into similar problem with HBase 0.90.2. My setting is 6 RSs
>> one master server, 3 ZK servers and about 20 clients (on different
>> servers) feeding the database with inserts. After few hours and around 2.5
>> millions rows inserted the process simply hangs with no what so ever error
>> indication (neither by ZK, master, regionservers nor the clients). All the
>> inserting clients stop virtually at the same moment, but HBase it self is
>> not down and can be queried.
>>
>> In fact, the clients not even crash on time out exception for its
>> connection. I have ran into such problem always when I attempted to run
>> such importing progress with the new HBase versions (even 0.90.1). Can
>> anybody address this problem? Does other have similar problems?
>>
>> I can provide further info about the configuration if needed.
>>
>> Stan Barton
>>
>>
>>
>>
>> ajay.gov wrote:
>>> Sorry, my server config was not attached. Its here:
>>> http://pastebin.com/U41QZGiq
>>>
>>> thanks
>>> -ajay
>>>
>>>
>>>
>>> ajay.gov wrote:
>>>> I am doing a load test for which I need to load a table with many rows.
>>>> I have a small java program that has a for loop and calls HTable.put.  I
>>>> am inserting a map of 2 items into a table that has one column family.
>>>> The limit of the for loop is currently 20000. However after 15876 rows
>>>> the call to Put hangs. I am using autoFlush on the HTable. Any ideas why
>>>> this may happen?
>>>>
>>>> The table configuration:
>>>> DESCRIPTION                                          ENABLED
>>>>   {NAME =>  'TABLE2', FAMILIES =>  [{NAME =>  'TABLE2_CF true
>>>>   1', BLOOMFILTER =>  'NONE', REPLICATION_SCOPE =>  '0'
>>>>   , COMPRESSION =>  'NONE', VERSIONS =>  '3', TTL =>  '2
>>>>   147483647', BLOCKSIZE =>  '65536', IN_MEMORY =>  'fal
>>>>   se', BLOCKCACHE =>  'true'}]}
>>>>
>>>> The HBase config on the client is the one in the hbase-default.xml. Some
>>>> values:
>>>> hbase.client.write.buffer=2097152
>>>> hbase.client.pause=1000
>>>> hbase.client.retries.number=10
>>>>
>>>> If i use another client I am able to put items to the table. I am also
>>>> able to scan items from the table using the hbase shell.
>>>>
>>>> I have attached the server configuratio
>>>> I don't see anything in the region server or master logs. I have them
>>>> here.
>>>>
>>>> The master server log:
>>>> 2011-04-06 19:02:40,149 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> RegionManager.rootScanner scanning meta region {server:
>>>> 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey:<>}
>>>> 2011-04-06 19:02:40,152 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
>>>> 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey:<>}
>>>> complete
>>>> 2011-04-06 19:02:40,157 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead,
>>>> average load 42.0
>>>> 2011-04-06 19:03:15,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> RegionManager.metaScanner scanning meta region {server:
>>>> 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey:<>}
>>>> 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> RegionManager.metaScanner scan of 40 row(s) of meta region {server:
>>>> 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey:<>}
>>>> complete
>>>> 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> All 1 .META. region(s) scanned
>>>>
>>>>
>>>> The region server logs:
>>>> 2011-04-06 19:02:21,294 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>>>> 2011-04-06 19:02:21,295 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>>>> sequenceid=1
>>>>
>>>>
>>>
>>

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com


Mime
View raw message