hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From OpenSource Dev <dev.opensou...@gmail.com>
Subject Hbase export failing in 0.90
Date Wed, 09 Oct 2013 21:38:58 GMT
Hi,
I'm trying to export 1 day data from HBase (version 0.90.4-cdh3u3),
but it fails every time with
'org.apache.hadoop.hbase.regionserver.LeaseException' error [1]

I increased the lease value, now it just take longer to fail. Command
that I used is:
sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.Export tsdb
/datatsdb/09-03 1 1378191600000 1378194600000

The export command starts ~7000 jobs, map mappers per node is set to
5. First ~200 task completes within 1-2 mins (each taking avg 6 sec)
but after that maps don't finish and eventually get the exception.
Ganglia shows increased in n/w throughout the period the MR job is
running. That suggests the map tasks are working, may be taking longer
to finish.

I've changed the lease settings (both on master and region servers):
hbase.regionserver.lease.period = 90000000
hbase.rpc.timeout = 90000000
hbase.client.scanner.caching = 10

Even with this high lease period (1500 Minutes), I'm getting
LeaseExeptions in 15 mins [2]

Any suggestion how I can resolve this issue ?

Thank you!


[1]

13/10/09 14:14:35 INFO mapred.JobClient: Task Id :
attempt_201310091100_0004_m_000540_2, Status : FAILED
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'2777638348379668022' does not exist
        at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1879)
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)


[2]

2013-10-09 13:14:07,341 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server data01.xx.xx.net/xx.xx.xx.xx:2181,
sessionid = 0x41825a012e95be, negotiated timeout = 40000
2013-10-09 13:29:09,141 INFO
org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-10-09 13:29:09,144 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-5530645372503000173' does not exist
at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1879)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

Mime
View raw message