hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Re: Help: RegionTooBusyException: failed to get a lock in 60000 ms
Date Fri, 05 Sep 2014 16:36:29 GMT
Hi Steven,

I did 1) and 2) and the error was during LoadIncrementalHFiles.

I can't do 3) because that CF is mostly used for mapreduce inputs, so a
continuous rowkey is preferred.

Jianshi



On Sat, Sep 6, 2014 at 12:29 AM, Magana-zook, Steven Alan <
maganazook1@llnl.gov> wrote:

> Jianshi,
>
> I have seen many solutions to importing this kind of data:
>
> 1. Pre-splitting regions (I did not try this)
>
> 2. Using a map reduce job to create HFiles instead of putting individual
> rows into the database
> (instructions here: http://hbase.apache.org/book/arch.bulk.load.html
>
> 3. Modifying the row key to not be monotonic
>
> I went with the third solution by pre-prending a random integer before the
> other fields in my composite row key ( "<random int>_<key field 1>_<key
> field 2>Š.")
>
> When you make any changes, you can verify it is working by viewing the
> Hbase web interface (port 60010 on the hbase master) to see the requests
> per second on the various region servers.
>
>
> Thank you,
> Steven Magana-Zook
>
>
>
>
>
>
> On 9/5/14 9:14 AM, "Jianshi Huang" <jianshi.huang@gmail.com> wrote:
>
> >Thanks Ted, I'll try to do a major compact.
> >
> >Hi Steven,
> >
> >Yes, most of my rows are hashed to make it randomly distributed, but one
> >column family has monotonically increasing rowkeys, and it's used for
> >recording sequence of events.
> >
> >Do you have a solution how to bulk import this kind of data?
> >
> >Jianshi
> >
> >
> >
> >On Sat, Sep 6, 2014 at 12:00 AM, Magana-zook, Steven Alan <
> >maganazook1@llnl.gov> wrote:
> >
> >> Hi Jianshi,
> >>
> >> What are the field(s) in your row key? If your row key is monotonically
> >> increasing then you will be sending all of your requests to one region
> >> server. Even after the region splits, all new entries will keep
> >>punishing
> >> one server (the region responsible for the split containing the new
> >>keys).
> >>
> >> See these articles that may help if this is indeed your issue:
> >> 1. http://hbase.apache.org/book/rowkey.design.html
> >> 2.
> >>
> >>
> http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-inc
> >>re
> >> asing-values-are-bad/
> >>
> >> Regards,
> >> Steven Magana-Zook
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 9/5/14 8:54 AM, "Jianshi Huang" <jianshi.huang@gmail.com> wrote:
> >>
> >> >Hi JM,
> >> >
> >> >What do you mean by the 'destination cluster'? The files are in the
> >>same
> >> >Hadoop/HDFS cluster where HBase is running.
> >> >
> >> >Do you mean do the bulk importing on HBase Master node?
> >> >
> >> >
> >> >Jianshi
> >> >
> >> >
> >> >On Fri, Sep 5, 2014 at 11:18 PM, Jean-Marc Spaggiari <
> >> >jean-marc@spaggiari.org> wrote:
> >> >
> >> >> Hi Jianshi,
> >> >>
> >> >> You might want to upload the file on the destination cluster first
> >>and
> >> >>then
> >> >> re-run your bulk load from there. That way the transfer time will
> >>not be
> >> >> taken into consideration for the timeout size the files will be
> >>local.
> >> >>
> >> >> JM
> >> >>
> >> >>
> >> >> 2014-09-05 11:15 GMT-04:00 Jianshi Huang <jianshi.huang@gmail.com>:
> >> >>
> >> >> > I'm importing 2TB of generated HFiles to HBase and I constantly
get
> >> >>the
> >> >> > following errors:
> >> >> >
> >> >> > Caused by:
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop
> >>>>.h
> >> >>base.RegionTooBusyException):
> >> >> > org.apache.hadoop.hbase.RegionTooBusyException: failed to get
a
> >>lock
> >> >>in
> >> >> > 60000 ms.
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>regionName=grapple_edges_v2,ff000000,1409817320781.6d2955c780b39523de73
> >>>>3f
> >> >>3565642d96.,
> >> >> > server=xxxxx.xxx.xxx,60020,1404854700728
> >> >> >         at
> >> >> >
> >>org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5851)
> >> >> >         at
> >> >> >
> >>org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5837)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.regionserver.HRegion.startBulkRegionOperation(H
> >>>>Re
> >> >>gion.java:5795)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.jav
> >>>>a:
> >> >>3543)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.jav
> >>>>a:
> >> >>3525)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegio
> >>>>nS
> >> >>erver.java:3277)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2
> >>>>.c
> >> >>allBlockingMethod(ClientProtos.java:28863)
> >> >> >         at
> >> >> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
> >> >> >         at
> >> >>org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcSc
> >>>>he
> >> >>duler.java:160)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcSche
> >>>>du
> >> >>ler.java:38)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler
> >>>>.j
> >> >>ava:110)
> >> >> >         at java.lang.Thread.run(Thread.java:724)
> >> >> >
> >> >> >         at
> >> >> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1498)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java
> >>>>:1
> >> >>684)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.
> >>>>ca
> >> >>llBlockingMethod(RpcClient.java:1737)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$B
> >>>>lo
> >> >>ckingStub.bulkLoadHFile(ClientProtos.java:29276)
> >> >> >         at
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUti
> >>>>l.
> >> >>java:1548)
> >> >> >         ... 11 more
> >> >> >
> >> >> >
> >> >> > What makes the region too busy? Is there a way to improve it?
> >> >> >
> >> >> > Does that also mean some part of my data are not correctly
> >>imported?
> >> >> >
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > --
> >> >> > Jianshi Huang
> >> >> >
> >> >> > LinkedIn: jianshi
> >> >> > Twitter: @jshuang
> >> >> > Github & Blog: http://huangjs.github.com/
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >Jianshi Huang
> >> >
> >> >LinkedIn: jianshi
> >> >Twitter: @jshuang
> >> >Github & Blog: http://huangjs.github.com/
> >>
> >>
> >
> >
> >--
> >Jianshi Huang
> >
> >LinkedIn: jianshi
> >Twitter: @jshuang
> >Github & Blog: http://huangjs.github.com/
>
>


-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message