Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAEo-6+TF4im5nOiUZudjAaN1jgNxFx6jyFL9VfFug4s-UuekCQ@mail.gmail.com>
References: 
 <CA+fHtaxaaXuzfiny5c4FRokAP0u-MpbsV5BdRBZEx-wCaop7xQ@mail.gmail.com>
	<CAEo-6+TF4im5nOiUZudjAaN1jgNxFx6jyFL9VfFug4s-UuekCQ@mail.gmail.com>
Date: Sun, 28 Jun 2015 23:09:15 +0530
Message-ID: 
 <CA+fHtawCmf-6k+UmdmtGmXojtnzoRLkUBUxcphjO9MSwnwMFig@mail.gmail.com>
Subject: Re: Static variable in reducer
From: Ravikant Dindokar <ravikant.iisc@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11c25ea0b9fe910519977373

--001a11c25ea0b9fe910519977373
Content-Type: text/plain; charset=UTF-8

Thanks a lot Shahab.

On Sun, Jun 28, 2015 at 8:38 PM, Shahab Yunus <shahab.yunus@gmail.com>
wrote:

> You asked a similar question earlier also so I will copy those comments
> here with what I replied then:
>
> http://hadoop-common.472056.n3.nabble.com/how-to-assign-unique-ID-Long-Value-in-mapper-td4078062.html
> Basically, to summarize, you shouldn't incorporate common sharable state
> among reducers. You need to rethink your design.
>
> Moving on, if you still want to do this then in your scenario: If a
> reducer fails (runs out of memory, hdd crashes etc. or in case of
> speculative execution,) then it will be given a new attempt number with the
> old task id when it is being recomputed/retried and your custom counter
> variable (the static one) should be reinitialized (as it will be in a new
> JVM).
>
> Regards,
> Shahab
>
> On Sun, Jun 28, 2015 at 6:45 AM, Ravikant Dindokar <
> ravikant.iisc@gmail.com> wrote:
>
>> Hi Hadoop user,
>>
>> I have graph data file in the form of edge list
>> <Source Vertex_id> <Sink Vertex_id>
>>
>> I want to assign each edge a unique ID. In the map function I emit
>> (key,value) as (<Source Vertex_id>, <Sink Vertex_id>)
>>
>> In the reducer, for each value , I am using a combination of static count
>> variable, and task id (context.getTaskAttemptID().getTaskID().getId()) to
>> generate a unique ID.
>>
>> edgeId=(localcount <<16)|(taskId << 55);
>>
>> I am able to generate unique IDs.
>>
>> My question is if a reducer fails will this work?
>>
>> What exactly happens when a reducer fails and computed again?
>>
>> PFA source code for mapper & reducer.
>>
>> Thanks
>> Ravikant
>>
>
>

--001a11c25ea0b9fe910519977373
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks a lot Shahab. <br></div><div class=3D"gmail_extra">=
<br><div class=3D"gmail_quote">On Sun, Jun 28, 2015 at 8:38 PM, Shahab Yunu=
s <span dir=3D"ltr">&lt;<a href=3D"mailto:shahab.yunus@gmail.com" target=3D=
"_blank">shahab.yunus@gmail.com</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">You asked a similar question earlier also so=
 I will copy those comments here with what I replied then:<div><a href=3D"h=
ttp://hadoop-common.472056.n3.nabble.com/how-to-assign-unique-ID-Long-Value=
-in-mapper-td4078062.html" target=3D"_blank">http://hadoop-common.472056.n3=
.nabble.com/how-to-assign-unique-ID-Long-Value-in-mapper-td4078062.html</a>=
 Basically, to summarize, you shouldn&#39;t incorporate common sharable sta=
te among reducers. You need to rethink your design.</div><div><div><div><br=
></div><div>Moving on, if you still want to do this then in your scenario: =
If a reducer fails (runs out of memory, hdd crashes etc. or in case of spec=
ulative execution,) then it will be given a new attempt number with the old=
 task id when it is being recomputed/retried and your custom counter variab=
le (the static one) should be reinitialized (as it will be in a new JVM).</=
div><div><br></div><div>Regards,</div><div>Shahab</div></div></div></div><d=
iv class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div c=
lass=3D"gmail_quote">On Sun, Jun 28, 2015 at 6:45 AM, Ravikant Dindokar <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:ravikant.iisc@gmail.com" target=3D"_bl=
ank">ravikant.iisc@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"=
gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-=
left:1ex"><div dir=3D"ltr"><div><div><div><div><div><div><div><div><div>Hi =
Hadoop user,<br><br></div>I have graph data file in the form of edge list<b=
r></div>&lt;Source Vertex_id&gt; &lt;Sink Vertex_id&gt;<br><br></div>I want=
 to assign each edge a unique ID. In the map function I emit (key,value) as=
 (&lt;Source Vertex_id&gt;, &lt;Sink Vertex_id&gt;)<br><br></div>In the red=
ucer, for each value , I am using a combination of static count variable, a=
nd task id (context.getTaskAttemptID().getTaskID().getId()) to generate a u=
nique ID. <br><br>edgeId=3D(localcount &lt;&lt;16)|(taskId &lt;&lt; 55);<br=
><br></div>I am able to generate unique IDs.<br><br></div>My question is if=
 a reducer fails will this work?=C2=A0 <br><br></div>What exactly happens w=
hen a reducer fails and computed again?<br><br></div><div>PFA source code f=
or mapper &amp; reducer.<br></div><div><br></div>Thanks<span><font color=3D=
"#888888"><br></font></span></div><span><font color=3D"#888888">Ravikant<br=
></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a11c25ea0b9fe910519977373--