hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiayu Ji <jiayu...@gmail.com>
Subject Re: Job fails while re attempting the task in multiple outputs case
Date Mon, 30 Dec 2013 19:57:03 GMT
I think if the task fails, the output related to that task will be clean up
before the second attempt. I am guessing you have this exception is because
you have two reducers tried to write to the same file. One thing you need
to be aware of is that all data that is supposed to be in the same file
should go to the same reducer. Let's say if you have data1 on reducer1 and
data2 on reducer2 and they are all going to be stored on fileAll, then you
will end up having that exception.


On Mon, Dec 30, 2013 at 11:22 AM, AnilKumar B <akumarb2010@gmail.com> wrote:

> Thanks Harsh.
>
> @Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
> I am using Apache Hadoop's multipleOutputs.
>
> But as you see in stack trace, it's not appending the attempt id to file
> name, it's only consists of task id.
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> Are you using the MultipleOutputs class shipped with Apache Hadoop or
>> one of your own?
>>
>> If its the latter, please take a look at gotchas to take care of
>> described at
>> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>>
>> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <akumarb2010@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am  using multiple outputs in our job. So whenever any reduce task
>> fails,
>> > all it's next task attempts are failing with file exist exception.
>> >
>> >
>> > The output file name should also append the task attempt right? But it's
>> > only appending the task id. Is this the bug or Some thing wrong from my
>> > side?
>> >
>> > Where should look in src code? I went through  code at
>> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
>> task
>> > id.
>> >
>> >
>> > Exception Trace:
>> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_0, Status : FAILED
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
>> > 13/12/29 09:15:04 INFO mapred.JobClient:  map 100% reduce 93%
>> > 13/12/29 09:15:23 INFO mapred.JobClient:  map 100% reduce 96%
>> > 13/12/29 09:17:31 INFO mapred.JobClient:  map 100% reduce 97%
>> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_1, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
>> > either because the filename is invalid or the file exists
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> >                 at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> >                 at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >                 at java.lang.reflect.Method.invoke(Method.java:597)
>> >                 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> >                 at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> >                 at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>> >                 at java.security.AccessController.doPrivileged(Native
>> > Method)
>> >                 at javax.security.auth.Subject.doAs(Subject.java:396)
>> >                 at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> >                 at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>> >
>> >                 at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>> >                 at
>> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> >                 at $Proxy7.create(Unknown Source)
>> >                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> >                 at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >                 at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >                 at java.lang.reflect.Method.invoke(Method.java:597)
>> >                 at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>> >                 at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>> >                 at $Proxy7.create(Unknown Source)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
>> >                 at
>> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
>> >                 at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>> >                 at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>> >                 at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>> >                 at
>> >
>> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
>> >                 at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
>> >                 at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
>> >                 at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
>> >                 at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
>> >                 at
>> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
>> >                 at
>> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>> >                 at
>> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>> >                 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> >                 at java.security.AccessController.doPrivileged(Native
>> > Method)
>> >                 at javax.security.auth.Subject.doAs(Subject.java:396)
>> >                 at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> >                 at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> >
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
>> > 13/12/29 09:19:36 INFO mapred.JobClient:  map 100% reduce 89%
>> > 13/12/29 09:19:54 INFO mapred.JobClient:  map 100% reduce 92%
>> > 13/12/29 09:20:11 INFO mapred.JobClient:  map 100% reduce 96%
>> > 13/12/29 09:22:52 INFO mapred.JobClient:  map 100% reduce 97%
>> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_2, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
>> either
>> > because the filename is invalid or the file exists
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> >                 at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> >                 at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> >                 at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >                 at java.lang.reflect.Method.invoke(Method.java:597)
>> >                 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> >                 at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> >
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>>
>>
>>
>> --
>> Harsh J
>>
>
>


-- 
Jiayu (James) Ji,

Cell: (312)823-7393

Mime
View raw message