On 17 June 2011 13:38, Harsh = J <harsh@clouder= a.com> wrote:

Geoffry,

The problem here is that the Reducer in Hadoop reuses the same
container object to pass on all values and keys. Thus, what you're
really holding in your second reducer's code are "References"= to this
object -> Which upon writing will all be a mess of duplicates and what not cause they are all gonna be referring to the last gotten value
every iteration.

The solution, when you want to persist a particular key or value
object, is to .clone() it into the list so that the list does store
real, new objects in it and not multiple references of the same
object.

On Sat, Jun 18, 2011 at 2:00 AM, Geoffry Roberts
<geoffry.roberts@gmail.com<= /a>> wrote:
> All,
>
> I have come across a situation that I don't understand.
>
> First Reducer:
>
> Behold the first of two reducers.=A0 A fragment of it's output fol= lows.
> Simple no?=A0 It doesn't do anything.=A0 I've highlighted two = records from the
> output.=A0 Keep them in mind.=A0 Now lets look at the second reducer.<= br> >
> protected void reduce(Text key, Iterable<Text> visitors, Context= ctx)
> =A0throws IOException, InterruptedException {
> =A0=A0=A0 for (Text visitor : visitors) {
> =A0=A0 =A0=A0=A0 ctx.write(key, visitor);
> =A0=A0=A0 }
> =A0}
>
> 2005-09-16=3D33614=A0=A0=A0 42340108=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D33614=A0=A0=A0 42340106=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D33614=A0=A0=A0 42340113=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D44135=A0=A0=A0 42324490=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D44135=A0=A0=A0 42339700=A0=A0=A0 more=3D=3D>
> ...
> 2005-09-16=3D44135=A0=A0=A0 42324489=A0=A0=A0 more=3D=3D>
>
>
> Second Reducer:
>
> This is a variation on the reducer from above.=A0 A fragment of it'= ;s output
> follows.=A0 The difference is I add all visitors to a list then I iter= ate
> through the list to produce my output.=A0 Remember the two highlighted= records
> from above? They are now showing up in the output as duplicates and th= e
> other records appear to be missing.=A0 Why?=A0 I have never seen an Ar= rayList
> behave like this.=A0 It must have something to do with hadoop.
>
> I have a reasons for using the list.=A0 One such reason is that I must= have a
> full count of all visitors before I can do my output, but I spare you.=
>
> To my mind, this second reducer should output the same as the first. >
> protected void reduce(Text key, Iterable<Text> visitors, Context= ctx)
> throws IOException, InterruptedException {
> =A0=A0=A0 List<Text> list =3D new ArrayList<Text>();
> =A0=A0=A0 for (Text visitor : visitors) {
> =A0=A0=A0 =A0=A0=A0 list.add(visitor);
> =A0=A0=A0 }
> =A0=A0=A0 for (Text visitor : list) {
> =A0=A0=A0 =A0=A0=A0 ctx.write(key, visitor);
> =A0=A0=A0 }
> }
>
> 2005-09-16=3D33614=A0=A0=A0 42340113=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D33614=A0=A0=A0 42340113=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D33614=A0=A0=A0 42340113=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D44135=A0=A0=A0 42324489=A0=A0=A0 more=3D=3D>
> 2005-09-16=3D44135=A0=A0=A0 42324489=A0=A0=A0 more=3D=3D>
>
> Thanks in advance
>
> --
> Geoffry Roberts
>
>

--
Harsh J