hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravikant Dindokar <ravikant.i...@gmail.com>
Subject Static variable in reducer
Date Sun, 28 Jun 2015 10:45:14 GMT
Hi Hadoop user,

I have graph data file in the form of edge list
<Source Vertex_id> <Sink Vertex_id>

I want to assign each edge a unique ID. In the map function I emit
(key,value) as (<Source Vertex_id>, <Sink Vertex_id>)

In the reducer, for each value , I am using a combination of static count
variable, and task id (context.getTaskAttemptID().getTaskID().getId()) to
generate a unique ID.

edgeId=(localcount <<16)|(taskId << 55);

I am able to generate unique IDs.

My question is if a reducer fails will this work?

What exactly happens when a reducer fails and computed again?

PFA source code for mapper & reducer.


View raw message