Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jayunit100@gmail.com
 designates 209.85.223.175 as permitted sender)
References: 
 <CAKyGxMtWoEpSiUEZG_0gMbrP35q4nh0SBu909tkBCXJAgNdoXQ@mail.gmail.com>
 <CALr1C9o9uZ=S0qJMUQZuaMGE_32DhfAT+YCkKQ2QwcHkAZ=x1Q@mail.gmail.com>
 <CAKyGxMtFjFkdP2ZycJgYDcCJH3jrDPJQukxuY5APjFqCftgJng@mail.gmail.com>
 <CALr1C9oBG80-bx6093qc9NQxhSm6L1DJ2mnVsz3rqBFGEgFwcw@mail.gmail.com>
 <CAOcnVr1684WtqTo9TbULiYAC-QergGopv21pahDG=X_-HyV5QQ@mail.gmail.com>
Mime-Version: 1.0 (1.0)
In-Reply-To: 
 <CAOcnVr1684WtqTo9TbULiYAC-QergGopv21pahDG=X_-HyV5QQ@mail.gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-33F8C5CB-DF31-4D1B-960B-9DD25F79E658
Content-Transfer-Encoding: 7bit
Message-Id: <FDE2FCF3-CC24-4DBB-B52B-2F8B5ECAF249@gmail.com>
Cc: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
From: Jay Vyas <jayunit100@gmail.com>
Subject: Re: Why LineRecordWriter.write(..) is synchronized
Date: Thu, 8 Aug 2013 07:53:07 -0400
To: "user@hadoop.apache.org" <user@hadoop.apache.org>


--Apple-Mail-33F8C5CB-DF31-4D1B-960B-9DD25F79E658
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Then is this a bug?  Synchronization in absence of any race condition is nor=
mally considered "bad".

In any case id like to know why this writer is synchronized whereas the othe=
r one are not.. That is, I think, then point at issue: either other writers s=
hould be synchronized or else this one shouldn't be - consistency across the=
 write implementations is probably desirable so that changes to output forma=
ts or record writers don't lead to bugs in multithreaded environments .

On Aug 8, 2013, at 6:50 AM, Harsh J <harsh@cloudera.com> wrote:

> While we don't fork by default, we do provide a MultithreadedMapper implem=
entation that would require such synchronization. But if you are asking is i=
t necessary, then perhaps the answer is no.
>=20
> On Aug 8, 2013 3:43 PM, "Azuryy Yu" <azuryyyu@gmail.com> wrote:
>> its not hadoop forked threads, we may create a line record writer, then c=
all this writer concurrently.
>>=20
>> On Aug 8, 2013 4:00 PM, "Sathwik B P" <sathwik.bp@gmail.com> wrote:
>>> Hi,
>>> Thanks for your reply.
>>> May I know where does hadoop fork multiple threads to use a single Recor=
dWriter.
>>>=20
>>> regards,
>>> sathwik
>>>=20
>>> On Thu, Aug 8, 2013 at 7:06 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>> because we may use multi-threads to write a single file.
>>>>=20
>>>> On Aug 8, 2013 2:54 PM, "Sathwik B P" <sathwik@apache.org> wrote:
>>>>> Hi,
>>>>>=20
>>>>> LineRecordWriter.write(..) is synchronized. I did not find any other R=
ecordWriter implementations define the write as synchronized.
>>>>> Any specific reason for this.
>>>>>=20
>>>>> regards,
>>>>> sathwik

--Apple-Mail-33F8C5CB-DF31-4D1B-960B-9DD25F79E658
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: 7bit

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Then is this a bug? &nbsp;Synchronization in absence of any race condition is normally considered "bad".<br></div><div><br></div><div>In any case id like to know why this writer is synchronized whereas the other one are not.. That is, I think, then point at issue: either other writers should be synchronized or else this one shouldn't be - consistency across the write implementations is probably desirable so that changes to output formats or record writers don't lead to bugs in multithreaded environments .</div><div><br>On Aug 8, 2013, at 6:50 AM, Harsh J &lt;<a href="mailto:harsh@cloudera.com">harsh@cloudera.com</a>&gt; wrote:<br><br></div><blockquote type="cite"><div><p dir="ltr">While we don't fork by default, we do provide a MultithreadedMapper implementation that would require such synchronization. But if you are asking is it necessary, then perhaps the answer is no.</p>
<div class="gmail_quote">On Aug 8, 2013 3:43 PM, "Azuryy Yu" &lt;<a href="mailto:azuryyyu@gmail.com">azuryyyu@gmail.com</a>&gt; wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">its not hadoop forked threads, we may create a line record writer, then call this writer concurrently.</p>
<div class="gmail_quote">On Aug 8, 2013 4:00 PM, "Sathwik B P" &lt;<a href="mailto:sathwik.bp@gmail.com" target="_blank">sathwik.bp@gmail.com</a>&gt; wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

Hi,<br>Thanks for your reply.<br>May I know where does hadoop fork multiple threads to use a single RecordWriter.<br><br>regards,<br>sathwik<br><br><div class="gmail_quote">On Thu, Aug 8, 2013 at 7:06 AM, Azuryy Yu <span dir="ltr">&lt;<a href="mailto:azuryyyu@gmail.com" target="_blank">azuryyyu@gmail.com</a>&gt;</span> wrote:<br>


<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">because we may use multi-threads to write a single file.</p><div><div>
<div class="gmail_quote">On Aug 8, 2013 2:54 PM, "Sathwik B P" &lt;<a href="mailto:sathwik@apache.org" target="_blank">sathwik@apache.org</a>&gt; wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


Hi,<br><br>LineRecordWriter.write(..) is synchronized. I did not find any other RecordWriter implementations define the write as synchronized.<br>Any specific reason for this.<br><br>regards,<br>sathwik<br>
</blockquote></div>
</div></div></blockquote></div><br>
</blockquote></div>
</blockquote></div>
</div></blockquote></body></html>
--Apple-Mail-33F8C5CB-DF31-4D1B-960B-9DD25F79E658--