Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of aagarwal@hortonworks.com
 designates 209.85.219.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANXCz3T9jeeAg505bdnZCO-Ow_3uRgTt+sxEAJ3Wrwg93KOMkQ@mail.gmail.com>
References: 
 <CANXCz3T9jeeAg505bdnZCO-Ow_3uRgTt+sxEAJ3Wrwg93KOMkQ@mail.gmail.com>
Date: Thu, 10 Jul 2014 11:00:17 -0700
Message-ID: 
 <CAFiYk=q3yjCdX+t1CwVVrJHZZodDnd8mav7Ryk-HrNyXagC+3g@mail.gmail.com>
Subject: Re: multiple map tasks writing in same hdfs file -issue
From: Arpit Agarwal <aagarwal@hortonworks.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e01184956f9d54604fdda9889

--089e01184956f9d54604fdda9889
Content-Type: text/plain; charset=UTF-8

HDFS is single-writer, multiple-reader (see sec 8.3.1 of
http://aosabook.org/en/hdfs.html). You cannot have multiple writers for a
single file at a time.


On Thu, Jul 10, 2014 at 2:55 AM, rab ra <rabmdu@gmail.com> wrote:

> Hello
>
>
> I have one use-case that spans multiple map tasks in hadoop environment. I
> use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
> into a file stored in hdfs. This file is shared across all the map tasks.
> Though, they all computes thier output but some of them are missing in the
> output file.
>
>
>
> The output file is an excel file with 8 parameters(headings). Each map
> task is supposed to compute all these 8 values, and save it as soon as it
> is computed. This means, the programming logic of a map task opens the
> file, writes the value and close, 8 times.
>
>
>
> Can someone give me a hint on whats going wrong here?
>
>
>
> Is it possible to make more than one map task to write in a shared file in
> HDFS?
>
> Regards
> Rab
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

--089e01184956f9d54604fdda9889
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>HDFS is single-writer, multiple-reader (see sec 8.3.1=
 of <a href=3D"http://aosabook.org/en/hdfs.html">http://aosabook.org/en/hdf=
s.html</a>). You cannot have multiple writers for a single file at a  time.=
<br>
</div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">O=
n Thu, Jul 10, 2014 at 2:55 AM, rab ra <span dir=3D"ltr">&lt;<a href=3D"mai=
lto:rabmdu@gmail.com" target=3D"_blank">rabmdu@gmail.com</a>&gt;</span> wro=
te:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hello<br>=C2=A0<br><br>I have one use-case t=
hat spans multiple map tasks in hadoop environment. I use hadoop 1.2.1 and =
with 6 task nodes. Each map task writes their output into a file stored in =
hdfs. This file is shared across all the map tasks. Though, they all comput=
es thier output but some of them are missing in the output file.<br>

<br>=C2=A0<br><br>The output file is an excel file with 8 parameters(headin=
gs). Each map task is supposed to compute all these 8 values, and save it a=
s soon as it is computed. This means, the programming logic of a map task o=
pens the file, writes the value and close, 8 times.<br>

<br>=C2=A0<br><br>Can someone give me a hint on whats going wrong here?<br>=
<br>=C2=A0<br><br>Is it possible to make more than one map task to write in=
 a shared file in HDFS?<br><br>Regards<span class=3D"HOEnZb"><font color=3D=
"#888888"><br>
Rab<br>
</font></span></blockquote></div><br></div>

<br>
<span style=3D"color:rgb(128,128,128);font-family:Arial,sans-serif;font-siz=
e:10px">CONFIDENTIALITY NOTICE</span><br style=3D"color:rgb(128,128,128);fo=
nt-family:Arial,sans-serif;font-size:10px"><span style=3D"color:rgb(128,128=
,128);font-family:Arial,sans-serif;font-size:10px">NOTICE: This message is =
intended for the use of the individual or entity to which it is addressed a=
nd may contain information that is confidential, privileged and exempt from=
 disclosure under applicable law. If the reader of this message is not the =
intended recipient, you are hereby notified that any printing, copying, dis=
semination, distribution, disclosure or forwarding of this communication is=
 strictly prohibited. If you have received this communication in error, ple=
ase contact the sender immediately and delete it from your system. Thank Yo=
u.</span>
--089e01184956f9d54604fdda9889--