hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Magana-zook, Steven Alan" <maganazo...@llnl.gov>
Subject Re: Help: RegionTooBusyException: failed to get a lock in 60000 ms
Date Fri, 05 Sep 2014 16:29:19 GMT
Jianshi,

I have seen many solutions to importing this kind of data:

1. Pre-splitting regions (I did not try this)

2. Using a map reduce job to create HFiles instead of putting individual
rows into the database
(instructions here: http://hbase.apache.org/book/arch.bulk.load.html

3. Modifying the row key to not be monotonic

I went with the third solution by pre-prending a random integer before the
other fields in my composite row key ( "<random int>_<key field 1>_<key
field 2>Š.")

When you make any changes, you can verify it is working by viewing the
Hbase web interface (port 60010 on the hbase master) to see the requests
per second on the various region servers.


Thank you,
Steven Magana-Zook






On 9/5/14 9:14 AM, "Jianshi Huang" <jianshi.huang@gmail.com> wrote:

>Thanks Ted, I'll try to do a major compact.
>
>Hi Steven,
>
>Yes, most of my rows are hashed to make it randomly distributed, but one
>column family has monotonically increasing rowkeys, and it's used for
>recording sequence of events.
>
>Do you have a solution how to bulk import this kind of data?
>
>Jianshi
>
>
>
>On Sat, Sep 6, 2014 at 12:00 AM, Magana-zook, Steven Alan <
>maganazook1@llnl.gov> wrote:
>
>> Hi Jianshi,
>>
>> What are the field(s) in your row key? If your row key is monotonically
>> increasing then you will be sending all of your requests to one region
>> server. Even after the region splits, all new entries will keep
>>punishing
>> one server (the region responsible for the split containing the new
>>keys).
>>
>> See these articles that may help if this is indeed your issue:
>> 1. http://hbase.apache.org/book/rowkey.design.html
>> 2.
>> 
>>http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-inc
>>re
>> asing-values-are-bad/
>>
>> Regards,
>> Steven Magana-Zook
>>
>>
>>
>>
>>
>>
>> On 9/5/14 8:54 AM, "Jianshi Huang" <jianshi.huang@gmail.com> wrote:
>>
>> >Hi JM,
>> >
>> >What do you mean by the 'destination cluster'? The files are in the
>>same
>> >Hadoop/HDFS cluster where HBase is running.
>> >
>> >Do you mean do the bulk importing on HBase Master node?
>> >
>> >
>> >Jianshi
>> >
>> >
>> >On Fri, Sep 5, 2014 at 11:18 PM, Jean-Marc Spaggiari <
>> >jean-marc@spaggiari.org> wrote:
>> >
>> >> Hi Jianshi,
>> >>
>> >> You might want to upload the file on the destination cluster first
>>and
>> >>then
>> >> re-run your bulk load from there. That way the transfer time will
>>not be
>> >> taken into consideration for the timeout size the files will be
>>local.
>> >>
>> >> JM
>> >>
>> >>
>> >> 2014-09-05 11:15 GMT-04:00 Jianshi Huang <jianshi.huang@gmail.com>:
>> >>
>> >> > I'm importing 2TB of generated HFiles to HBase and I constantly get
>> >>the
>> >> > following errors:
>> >> >
>> >> > Caused by:
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop
>>>>.h
>> >>base.RegionTooBusyException):
>> >> > org.apache.hadoop.hbase.RegionTooBusyException: failed to get a
>>lock
>> >>in
>> >> > 60000 ms.
>> >> >
>> >> >
>> >>
>> 
>>>>regionName=grapple_edges_v2,ff000000,1409817320781.6d2955c780b39523de73
>>>>3f
>> >>3565642d96.,
>> >> > server=xxxxx.xxx.xxx,60020,1404854700728
>> >> >         at
>> >> > 
>>org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5851)
>> >> >         at
>> >> > 
>>org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5837)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.regionserver.HRegion.startBulkRegionOperation(H
>>>>Re
>> >>gion.java:5795)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.jav
>>>>a:
>> >>3543)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.jav
>>>>a:
>> >>3525)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegio
>>>>nS
>> >>erver.java:3277)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2
>>>>.c
>> >>allBlockingMethod(ClientProtos.java:28863)
>> >> >         at
>> >> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
>> >> >         at
>> >>org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcSc
>>>>he
>> >>duler.java:160)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcSche
>>>>du
>> >>ler.java:38)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler
>>>>.j
>> >>ava:110)
>> >> >         at java.lang.Thread.run(Thread.java:724)
>> >> >
>> >> >         at
>> >> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1498)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java
>>>>:1
>> >>684)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.
>>>>ca
>> >>llBlockingMethod(RpcClient.java:1737)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$B
>>>>lo
>> >>ckingStub.bulkLoadHFile(ClientProtos.java:29276)
>> >> >         at
>> >> >
>> >> >
>> >>
>> 
>>>>org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUti
>>>>l.
>> >>java:1548)
>> >> >         ... 11 more
>> >> >
>> >> >
>> >> > What makes the region too busy? Is there a way to improve it?
>> >> >
>> >> > Does that also mean some part of my data are not correctly
>>imported?
>> >> >
>> >> >
>> >> > Thanks,
>> >> >
>> >> > --
>> >> > Jianshi Huang
>> >> >
>> >> > LinkedIn: jianshi
>> >> > Twitter: @jshuang
>> >> > Github & Blog: http://huangjs.github.com/
>> >> >
>> >>
>> >
>> >
>> >
>> >--
>> >Jianshi Huang
>> >
>> >LinkedIn: jianshi
>> >Twitter: @jshuang
>> >Github & Blog: http://huangjs.github.com/
>>
>>
>
>
>-- 
>Jianshi Huang
>
>LinkedIn: jianshi
>Twitter: @jshuang
>Github & Blog: http://huangjs.github.com/


Mime
View raw message