hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Agarwal, Saurabh " <saurabh.agar...@citi.com>
Subject "this file is already being created by NN_Recovery" error
Date Mon, 20 Feb 2012 14:52:05 GMT
Hi,

I am getting the following error and hbase shell is hanging and we are not able to connect
hbase. Hadoop filesystem is fine and map reduce jobs are running ok.

Here is the warning message we are getting in the master log file.
for lease recovery on hdfs://<servername>:54310/user/root/hbase/.logs/<servername>,60020,1329596937708/<servername>%3A60020.1329607742925:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
failed to create file /user/root/hbase/.logs/<>,60020,1329596937708/<servername>%3A60020.1329607742925
for DFSClient_hb_m_:60000_1329688357813 on client <>, because this file is already being
created by NN_Recovery on <>
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1202)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:1157)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.recoverLease(NameNode.java:404)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955)

Any help is appreciated,
Thanks,
Saurabh.
-----Original Message-----
From: NNever [mailto:nneverwei@gmail.com]
Sent: Monday, February 20, 2012 9:04 AM
To: user@hbase.apache.org
Subject: HBase0.92: In Coprocessor's prePut method, muilt put on same HTable cause HBase crash

Hi~

On HBase0.92.

I write a coprosessor on table 'File'.In the prePut method,  I put several
new rows on File HTable instance. As u know , this op will trigger prePut
again.
I use this logic to realize something like "copy a File and auto copy its
all level's subfiles" .
The code seems no wrong, but everytime i trigger this coprocessor, the
HBase stuck and couldn't shutdown.
(through the log I found the coprocssor's code fall into dead-loop, but the
prePut's code don't have any place may cause dead-loop)

I try lot of times and finally get the reason: I cache the File's HTable
instance and use it like this

*private HTableInterface htFile = null;*
*...*
*@Override*
*public void prePut(...){*

*...*
*if(null == htFile){*
*   htFile = e.getEnvironment().getTable(Bytes.toBytes("File"));*
*}*
*...*
*htFile.put(some list of new puts);*
*...*

*}*

if I change the code and get HTable's instance everytime, the problem fixed.
But I want to know why* using the same Instance of HTable to put puts in
same prePut method may cause coprocessor dead-loop*?
(it just trigger prePut with same put again and again util the HBase
stuck...)

Thanks~

Mime
View raw message