hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Judd" <d...@zvents.com>
Subject hang during close of deleted file
Date Fri, 05 Dec 2008 15:18:49 GMT
I've run into a problem with Hadoop 0.19 where my process hangs when trying
to close an FSDataOutputStream after the file has gotten deleted out from
underneath it.

Here's the stack trace of the thread that is hanging:

"Reactor 0" prio=10 tid=0x00000000569dc000 nid=0x6a3b in Object.wait()
[0x0000000041d8a000..0x0000000041d8ab90]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3015)
    - locked <0x00002aaab3a01408> (a java.util.LinkedList)
    - locked <0x00002aaab3a00fd0> (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3104)
    - locked <0x00002aaab3a00fd0> (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3053)
    at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
    at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
    at
org.hypertable.DfsBroker.hadoop.OpenFileMap.RemoveAll(OpenFileMap.java:63)
    - locked <0x00002aaab39face0> (a
org.hypertable.DfsBroker.hadoop.OpenFileMap)
    at
org.hypertable.DfsBroker.hadoop.ConnectionHandler.handle(ConnectionHandler.java:137)
    at org.hypertable.AsyncComm.IOHandler.DeliverEvent(IOHandler.java:57)
    at
org.hypertable.AsyncComm.IOHandlerData.handle_disconnect(IOHandlerData.java:77)
    at org.hypertable.AsyncComm.IOHandlerData.run(IOHandlerData.java:118)
    at org.hypertable.AsyncComm.Reactor.run(Reactor.java:83)
    at java.lang.Thread.run(Thread.java:619)

And here are some of the exceptions that appear in my logs:

08/12/05 07:09:43 WARN hdfs.DFSClient: NotReplicatedYetException sleeping
/hypertable/servers/10.0.30.104_38060/log/range_txn/0.log retries left 2
08/12/05 07:09:44 INFO hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hypertable/servers/10.0.30.104_38060/log/range_txn/0.log File does not
exist. Holder DFSClient_1926295755 does not have any open files.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
    at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)

    at org.apache.hadoop.ipc.Client.call(Client.java:696)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
    at $Proxy0.addBlock(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy0.addBlock(Unknown Source)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

08/12/05 07:09:44 WARN hdfs.DFSClient: NotReplicatedYetException sleeping
/hypertable/servers/10.0.30.104_38060/log/range_txn/0.log retries left 1
08/12/05 07:09:48 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hypertable/servers/10.0.30.104_38060/log/range_txn/0.log File does not
exist. Holder DFSClient_1926295755 does not have any open files.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1331)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1322)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1250)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
    at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)

    at org.apache.hadoop.ipc.Client.call(Client.java:696)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
    at $Proxy0.addBlock(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy0.addBlock(Unknown Source)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2815)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2697)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

- Doug

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message