hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ooh Rong <oohr...@gmail.com>
Subject Re: MR hbase export is failing
Date Wed, 25 Jul 2012 02:00:46 GMT
We had a similar issue on CDH3u4(hbase 0.90) and upgrading to
CDH4(hbase 0.92) solved the problem for us.
If it's an option, I recommend upgrading HBase.


On Wed, Jul 25, 2012 at 4:04 AM, Paul Mackles <pmackles@adobe.com> wrote:
> I've seen this when writing when exporting to s3 and assumed it was
> related to write performance. We set hbase.regionserver.lease.period to
> the same values as the task timeout and it helped reduce the # of failures
> though we still get occasional task timeouts. I haven't seen this when
> writing to local HDFS though. Are any of your rows particularly wide? How
> loaded are your data nodes when running the export? Are you running too
> many mappers per node and starving hbase?
>
> On 7/24/12 2:52 PM, "Jeff Whiting" <jeffw@qualtrics.com> wrote:
>
>>What would cause a scanner timeout exception?  Is hdfs too slow?  Do I
>>just increase the scanner
>>timeout or is there a better approach.
>>
>>Thanks,
>>~Jeff
>>
>>running: hadoop jar /usr/lib/hbase/hbase-0.90.1-CDH3B4.jar export -D
>>dfs.replication=2 -D
>>mapred.output.compress=true -D
>>mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
>>-D
>>mapred.output.compression.type=BLOCK response /hbaseBackup/
>>partial/2012-07-20_2012-07-21/response 1 1342742400000 1342828800000
>>org.apache.hadoop.hbase.client.ScannerTimeoutException: 61294ms passed
>>since the last invocation,
>>timeout is currently set to 60000
>>         at
>>org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1114)
>>         at
>>org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(Table
>>RecordReaderImpl.java:143)
>>         at
>>org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableReco
>>rdReader.java:142)
>>         at
>>org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapT
>>ask.java:455)
>>         at
>>org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>>         at
>>org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>java:1115)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:234)
>>Caused by: org.apache.hadoop.hbase.UnknownScannerException:
>>org.apache.hadoop.hbase.UnknownScannerException: Name:
>>-3564751891935236449
>>         at
>>org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java
>>:1795)
>>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>>         at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at
>>org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>         at
>>org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>
>>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>Method)
>>         at
>>sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
>>essorImpl.java:39)
>>         at
>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr
>>uctorAccessorImpl.java:27)
>>         at
>>java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>         at
>>org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remot
>>eExceptionHandler.java:96)
>>         at
>>org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:8
>>3)
>>         at
>>org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:3
>>8)
>>         at
>>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>>n.getRegionServerWithRetries(HConnectionManager.java:1000)
>>         at
>>org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1100)
>>         ... 12 more
>>
>>--
>>Jeff Whiting
>>Qualtrics Senior Software Engineer
>>jeffw@qualtrics.com
>>
>

Mime
View raw message