hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tianying Chang <tych...@gmail.com>
Subject Re: export snapshot fail sometime due to LeaseExpiredException
Date Wed, 30 Apr 2014 17:08:03 GMT
The namenode has the same error. It seem the Exception are all for the
HFile that is bigger than 1G. Smaller HFile seems all fine.  Is it because
the default timeout too short for those copy task?

org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/ce168540863baee7b05b741775b38816/d/862ae8d7e1cd48d0a2b97658a816af8b
File does not exist. Holder DFSClient_NONMAPREDUCE_1731821456_1 does not
have any open files.

2014-04-30 16:49:51,923 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tychang (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.
LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/7dfa2b0afee391a7c65e097722ea237e/d/6bc9add98fe1402c8d556195ba6be8e9
File does not exist. Holder DFSClient_NONMAPREDUCE_224703696_1 does not
have any open files.

2014-04-30 16:49:51,923 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 25 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.complete from
10.183.170.97:52160: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
No lease on
/hbase/.archive/rich_pin_data_v1/7dfa2b0afee391a7c65e097722ea237e/d/6bc9add98fe1402c8d556195ba6be8e9
File does not exist. Holder DFSClient_NONMAPREDUCE_224703696_1 does not
have any open files.

org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/7dfa2b0afee391a7c65e097722ea237e/d/6bc9add98fe1402c8d556195ba6be8e9
File does not exist. Holder DFSClient_NONMAPREDUCE_224703696_1 does not
have any open files.

2014-04-30 16:51:09,548 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tychang (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.
LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/f1df459c3cd1be662133f9ae2b6f12cc/d/ccd80ef4569a41b0b84381014e5f7266
File does not exist. Holder DFSClient_NONMAPREDUCE_1186324301_1 does not
have any open files.

2014-04-30 16:51:09,548 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 47 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
10.180.161.116:48239: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
No lease on
/hbase/.archive/rich_pin_data_v1/f1df459c3cd1be662133f9ae2b6f12cc/d/ccd80ef4569a41b0b84381014e5f7266
File does not exist. Holder DFSClient_NONMAPREDUCE_1186324301_1 does not
have any open files.

org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/f1df459c3cd1be662133f9ae2b6f12cc/d/ccd80ef4569a41b0b84381014e5f7266
File does not exist. Holder DFSClient_NONMAPREDUCE_1186324301_1 does not
have any open files.

2014-04-30 16:53:31,996 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:tychang (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.
LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/93d87cd353d06fda40054553924f3554/d/b8b832530fe0413595357b291c4e9701
File does not exist. Holder DFSClient_NONMAPREDUCE_-1410010161_1 does not
have any open files.

2014-04-30 16:53:31,996 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 58 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
10.180.170.209:54236: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
No lease on
/hbase/.archive/rich_pin_data_v1/93d87cd353d06fda40054553924f3554/d/b8b832530fe0413595357b291c4e9701
File does not exist. Holder DFSClient_NONMAPREDUCE_-1410010161_1 does not
have any open files.

org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.archive/rich_pin_data_v1/93d87cd353d06fda40054553924f3554/d/b8b832530fe0413595357b291c4e9701
File does not exist. Holder DFSClient_NONMAPREDUCE_-1410010161_1 does not
have any open files.


On Wed, Apr 30, 2014 at 9:53 AM, Tianying Chang <tychang@gmail.com> wrote:

> we are using
>
> Hadoop 2.0.0-cdh4.2.0 and hbase 0.94.7. We also backported several
> snapshot related jira, e.g 10111(verify snapshot), 11083 (bandwidth
> throttle in exportSnapshot)
>
> I found when the  LeaseExpiredException first reported, that file indeed
> not there, and the map task retry. And I verifified couple minutes later,
> that HFile does exist under /.archive. But the retry map task still
> complain the same error of file  not exist...
>
> I will check the namenode log for the LeaseExpiredException.
>
>
> Thanks
>
> Tian-Ying
>
>
> On Wed, Apr 30, 2014 at 9:33 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Can you give us the hbase and hadoop releases you're using ?
>>
>> Can you check namenode log around the time LeaseExpiredException was
>> encountered ?
>>
>> Cheers
>>
>>
>> On Wed, Apr 30, 2014 at 9:20 AM, Tianying Chang <tychang@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > When I export large table with 460+ regions, I saw the exportSnapshot
>> job
>> > fail sometime (not all the time). The error of the map task is below:
>> But I
>> > verified the file highlighted below, it does exist. Smaller table seems
>> > always pass. Any idea? Is it because it is too big and get session
>> timeout?
>> >
>> >
>> >
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
>> > No lease on
>> >
>> /hbase/.archive/rich_pin_data_v1/7713d5331180cb610834ba1c4ebbb9b3/d/eef3642f49244547bb6606d4d0f15f1f
>> > File does not exist. Holder DFSClient_NONMAPREDUCE_279781617_1 does
>> > not have any open files.
>> >         at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2396)
>> >         at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2387)
>> >         at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2183)
>> >         at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:481)
>> >         at
>> >
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
>> >         at
>> >
>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
>> >         at org.apache.hadoop.ipc.ProtobufR
>> >
>> >
>> >
>> > Thanks
>> >
>> > Tian-Ying
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message