hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashwanth Kumar <ashwanthku...@googlemail.com>
Subject HBase Export MR - Some mappers getting Stuck
Date Tue, 30 Apr 2013 05:42:51 GMT
Hey,

I have this issue where in some mappers get stuck mid-way while running an
HBase Export.

jstack on the Task gives me this --

"main" prio=10 tid=0x00007f535800a800 nid=0x1c72 in Object.wait()
[0x00007f535f5e4000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000eb1d6598> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
 at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:904)
 - locked <0x00000000eb1d6598> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
 at $Proxy7.next(Unknown Source)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:80)
 at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
 at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
at
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:133)
 at
org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:142)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
 at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)

If I fail / kill the attempt once, the task get's completed without any
issues.

I am not able to replicate the issue all the time, but happens at frequent
intervals though.

HBase Version - 0.94.2
Hadoop Version - 1.0.4
Client Scanner Caching - 500

Also when I check the counters, I have some multiple of 500 records as Map
Input Records.

Any insight to why this could be happening?


-- 

Ashwanth Kumar / ashwanthkumar.in

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message