Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Ashish Paliwal <ashishpaliwal83@gmail.com>
Date: Wed, 21 Dec 2016 18:28:44 +0530
Message-ID: <CAHumUzgsobhapsVLpB+2AJG4raW3wWd0TBtFWbAicykShbA6Zg@mail.gmail.com>
Subject: Hadoop MultiOutputs API Issue
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11376f927e363605442ab65a
archived-at: Wed, 21 Dec 2016 12:58:57 -0000

--001a11376f927e363605442ab65a
Content-Type: text/plain; charset=UTF-8

Hi,

Hadoop Map Reduce version: 2.2.0

We are using MultiOutputs to write mullitple output files from Mapper(No
reducer). As per requirement, multioutput should write in directory other
than job's default output directory. So We used below MultiOutput method to
write in different directory.

 public <K, V> void
<http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.action/0.2.7/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#>
write(String
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String>
 namedOutput, K key, V value,String
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String>
 baseOutputPath)

Now, if any Map task run for longer time, then (cause speculative execution
enabled), hadoop start parallel task to complete task early. Now, both task
trying to write in same directory in same file. Second task failed with
"File already exists issue" and so Job.

After analyzing it founds that, like default context writer, *MultiOutputs
API does not create any temporary directory*. It directly starts writing
into output directory. and the reason is FileOutputCommitter used by
default context writer (and so Application Master) is different
than MultiOutputs.writer. So in case of MultiOutput, none of the method of
FileOutputCommitter is get called.

So is it known issue or default behavior? And what is the solution for this
problem?


Regards,
Ashish.

--001a11376f927e363605442ab65a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,</div><div><br></div><div>Hadoop Map Reduce versio=
n: 2.2.0</div><div><br></div><div>We are using MultiOutputs to write mullit=
ple output files from Mapper(No reducer). As per requirement, multioutput s=
hould write in directory other than job&#39;s default output directory. So =
We used below MultiOutput=C2=A0method to write in different directory.=C2=
=A0</div><div><br></div><div>=C2=A0<span class=3D"gmail-kw">public</span>=
=C2=A0&lt;K,=C2=A0V&gt;=C2=A0<span class=3D"gmail-kw">void</span>=C2=A0<a c=
lass=3D"gmail-context-menu" href=3D"http://grepcode.com/file/repo1.maven.or=
g/maven2/com.ning/metrics.action/0.2.7/org/apache/hadoop/mapreduce/lib/outp=
ut/MultipleOutputs.java#"><img width=3D"11" height=3D"16" title=3D"More ...=
" class=3D"gmail-img_base gmail-img_jdt_elcl16_thin_show_toolbar" src=3D"ht=
tp://grepcode.com/static/app/images/1x1.gif" border=3D"0"></a><a name=3D"Mu=
ltipleOutputs.write%28java.lang.String%2Cjava.lang.Object%2Cjava.lang.Objec=
t%2Cjava.lang.String%29">write</a>(<a title=3D"java.lang.String" href=3D"ht=
tp://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/=
java/lang/String.java#String"><font color=3D"#0066cc">String</font></a>=C2=
=A0<span class=3D"gmail-mark-65#0">namedOutput</span>,=C2=A0K=C2=A0<span cl=
ass=3D"gmail-mark-66#0">key</span>,=C2=A0V=C2=A0<span class=3D"gmail-mark-6=
7#0">value</span>,<a title=3D"java.lang.String" href=3D"http://grepcode.com=
/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.=
java#String"><font color=3D"#0066cc">String</font></a>=C2=A0<span class=3D"=
gmail-mark-68#0" style=3D"background:rgb(255,187,136)">baseOutputPath</span=
>)</div><div><br></div><div>Now, if any Map task run for longer time, then =
(cause=C2=A0speculative execution enabled), hadoop start parallel task to c=
omplete task early. Now, both task trying to write in same directory in sam=
e file. Second task failed with &quot;File already exists issue&quot; and s=
o Job. </div><div><br></div><div>After analyzing it founds that, like defau=
lt context writer,=C2=A0<strong>MultiOutputs API does not create any tempor=
ary directory</strong>. It directly starts writing into output directory.=
=C2=A0and the reason is FileOutputCommitter used by default context writer =
(and so Application Master) is different than=C2=A0MultiOutputs.writer. So =
in case of MultiOutput, none of the method of FileOutputCommitter is get ca=
lled.</div><div>=C2=A0</div><div>So is it known issue or default behavior? =
And what is the solution=C2=A0for this problem?</div><div><br></div><div><b=
r></div><div>Regards,</div><div>Ashish.</div></div>

--001a11376f927e363605442ab65a--