hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudharsan Sampath <sudha...@gmail.com>
Subject HDFS error : Could not Complete file
Date Fri, 04 Nov 2011 09:17:01 GMT
Hi,

I have a simple map-reduce program [map only :) ]that reads the input and
emits the same to n outputs on a single node cluster with max map tasks set
to 10 on a 16 core processor machine.

After a while the tasks begin to fail with the following exception log.

2011-01-01 03:17:52,149 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=temp,temp
    ip=/x.x.x.x cmd=delete
 src=/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2
  dst=null        perm=null
2011-01-01 03:17:52,156 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: addStoredBlock request received for
blk_7046642930904717718_23143 on x.x.x.x:<port> size 66148 But it does not
belong to any file.
2011-01-01 03:17:52,156 WARN org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: failed to complete
/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000
because dir.getFileBlocks() is null  and pendingFile is null
2011-01-01 03:17:52,156 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 12 on 9000, call
complete(/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000,
DFSClient_attempt_201101010256_0006_m_000000_2) from x.x.x.x:<port> error:
java.io.IOException: Could not complete write to file
/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000
by DFSClient_attempt_201101010256_0006_m_000000_2
java.io.IOException: Could not complete write to file
/TestMultipleOuputs1320394241986/_temporary/_attempt_201101010256_0006_m_000000_2/Output0-m-00000
by DFSClient_attempt_201101010256_0006_m_000000_2
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497)
        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962)


Looks like there's a delete command issued by FsNameSystem.audit before the
it errors out stating it could not complete write to the file inside that..

Any clue on what could have gone wrong?

Thanks
Sudharsan S

Mime
View raw message