hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: copyTable mapreduce job failing with ScannerTimeoutException
Date Fri, 09 Sep 2016 20:26:07 GMT
The 'Above memstore limit' warning meant that your region server(s) was
under pressure of the write load.

Can you share memstore related config parameters ?

Did you observe hot spotting in the region server(s) ?

Cheers

On Fri, Sep 9, 2016 at 1:20 PM, marjana <mivkovic@us.ibm.com> wrote:

> Looked at regionserver logs and found different errors:
>
> 2016-09-09 14:52:26,172 WARN  [hconnection-0x41059e91-
> shared--pool3-t500214]
> client.AsyncProcess: #2, table=3079_01, attempt=5/4 failed=1ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=3079_01,aeeeeee4,1473428857901.3042d01acd1bd7066934a3374ce1a7
> 4a.,
> server=hb01dn25,16020,1471966355932, memstoreSize=537147864,
> blockingMemStoreSize=536870912
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:
> 3630)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.
> batchMutate(HRegion.java:2858)
> .....
> Fri Sep 09 14:52:24 UTC 2016; not retrying 1 - final failure
> 2016-09-09 14:52:26,172 ERROR
> [ReplicationRpcServer.handler=0,queue=0,port=16020]
> regionserver.ReplicationSink: Unable to accept edit because:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed
> 1 action: RegionTooBusyException: 1 time,
>
>
> My CopyTable ended some 3 hours later (with the error it failed). I wonder
> why it hasn't errored out at 14:52.
> Will try setting hbase.regionserver.lease.period and hbase.rpc.timeout to
> double the default, 120 seconds on the target cluster.
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.
> nabble.com/copyTable-mapreduce-job-failing-with-ScannerTimeoutException-
> tp4082544p4082563.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message