hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: HBase bulk load failing with RegionTooBusyException
Date Tue, 25 Mar 2014 22:03:16 GMT
Can you check region server log to see if there was concurrent compaction
going on for table50,\x05\x80\x00\x00\x00\x00\x1E\x84\x80\
xDF\x00\x00\x01D\xFD\xFD\x06\x7F^\xFF\xFF\xFF\xFD,1395709055737.
5a772656accd69dd0354a09fc0c4415f. ?

If so, how long did the compaction take ?

Cheers


On Tue, Mar 25, 2014 at 2:52 PM, Mahesh Garg <m_garg@hotmail.com> wrote:

> Hi,
>
> We are trying to bulk load into HBase. But, it it failing. We are using a
> very standard process:
>
>       1. Create HFiles using an MR job.
>       2. Call "completebulkload" to move HFiles into regions.
>
> The table is pre-split. We are using HBase version "0.94.15-cdh4.6.0".
>
> The load process runs in a loop and, about 10 million rows are loaded in
> each attempt. However, the process fails in
> "completebulkload" with RegionTooBusyException. This error happen at
> different times after loading about 120 M rows.
>
> Any suggestion on how to avoid/debug this problem? Any parameters to tune?
> The stack track is listed below.
>
> Thanks for your help!
> Garg
>
>
> 14/03/25 03:10:39 INFO mapreduce.LoadIncrementalHFiles: Trying to load
> hfile=hdfs://XXX.XXX.XXX.XXX:8020/user/userName/outputdir/cf2/e2680aee969a47ae97c6980bbd76ea66
> first=\x06\x00\x00\x00\x00\x00\x00\x00\x07_0_\x00\x00\x01D\xF73\xB7\x1F_\x00\x00\x00\x01
> last=\x06\x00\x00\x00\x00\x00w[B_25_\x00\x00\x01D\xF74\xD7\xAC_\x00\x00\x00\x01
> 14/03/25 03:21:50 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> unrecoverable error from region server
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=10, exceptions:
> Tue Mar 25 03:11:39 UTC 2014,
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@29324479,
> org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in
> 60000 ms.
> regionName=table50,\x05\x80\x00\x00\x00\x00\x1E\x84\x80\xDF\x00\x00\x01D\xFD\xFD\x06\x7F^\xFF\xFF\xFF\xFD,1395709055737.5a772656accd69dd0354a09fc0c4415f.,
> server=XXX.XXX.XXX.XXX,60020,1395216882104
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5898)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5884)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.startBulkRegionOperation(HRegion.java:5843)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3640)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3623)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFiles(HRegionServer.java:2978)
>         at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRp
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message