hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AnilKumar B <akumarb2...@gmail.com>
Subject Job fails while re attempting the task in multiple outputs case
Date Mon, 30 Dec 2013 10:52:15 GMT
Hi,

I am  using multiple outputs in our job. So whenever any reduce task fails,
all it's next task attempts are failing with file exist exception.


The output file name should also append the task attempt right? But it's
only appending the task id. Is this the bug or Some thing wrong from my
side?

Where should look in src code? I went through  code at
FileOutputFormat$getTaskOutputPath(), but there it's only considering task
id.


Exception Trace:
13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
attempt_201312162255_60465_r_000008_0, Status : FAILED
13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
13/12/29 09:15:04 INFO mapred.JobClient:  map 100% reduce 93%
13/12/29 09:15:23 INFO mapred.JobClient:  map 100% reduce 96%
13/12/29 09:17:31 INFO mapred.JobClient:  map 100% reduce 97%
13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
attempt_201312162255_60465_r_000008_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
create /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client
10.103.10.31 either because the filename is invalid or the file exists
                at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
                at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
                at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
                at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
                at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
Source)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
                at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
                at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
                at java.security.AccessController.doPrivileged(Native
Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
                at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)

                at org.apache.hadoop.ipc.Client.call(Client.java:1118)
                at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
                at $Proxy7.create(Unknown Source)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
                at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
                at $Proxy7.create(Unknown Source)
                at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
                at
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
                at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
                at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
                at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
                at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
                at
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
                at
org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
                at
org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
                at
com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
                at
com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
                at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
                at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
                at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
                at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
                at java.security.AccessController.doPrivileged(Native
Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
                at org.apache.hadoop.mapred.Child.main(Child.java:249)

13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
13/12/29 09:19:36 INFO mapred.JobClient:  map 100% reduce 89%
13/12/29 09:19:54 INFO mapred.JobClient:  map 100% reduce 92%
13/12/29 09:20:11 INFO mapred.JobClient:  map 100% reduce 96%
13/12/29 09:22:52 INFO mapred.JobClient:  map 100% reduce 97%
13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
attempt_201312162255_60465_r_000008_2, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
create /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client
10.103.7.33 either because the filename is invalid or the file exists
                at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
                at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
                at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
                at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
                at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
Source)
                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
                at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)


Thanks & Regards,
B Anil Kumar.

Mime
View raw message