collector=2C Reporter reporter) throws IOException { . . = . GenericRecord record =3D null=3B for (GenericRecord value : values) { = . . = . record =3D value=3B record.put("rowkey"=2C key)=3B = . . = . collector.collect(record)=3B = }}=20 If I comment out the statement in red in the above code=2C the reduce funct= ion gets called properly with CORRECT key values pairs passed to reduce(). = However=2C if I add the statement in red to the routine=2C the reduce func= tion is called with WRONG key values pairs=2C in the sense that key2 paired= with values3=2C instead of values2=2C when passed to the reduce() routine.= I traced this problem by including Hadoop source code=2C such as ReduceTa= sk.java=2C Task.java=2C and Avro source code=2C such as HadoopReducer.java= =2C HadoopReducerBase.java=2C and all the serialization code. The problem = showed up on the second call of the reduce()=2C but I can not locate the ex= act place that cause the problem. My intuition is that this is incurred in= either the hadoop iterators after merge sort or Avro deserialization. Is = there anybody can help me on this? Thanks. Ey-Chih Chow = --_1047860a-22eb-43a6-9c3d-436bcd2c8380_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sorry I found that my previous message in the archive become all in black. = =3BLet me re-explain the problem. =3BThe following piece of code = for AvroReducer causes problem:

=3B =3B = =3B =3B =3B =3B public void reduce(Utf8 key=2C Iterable<= =3BGenericRecord>=3B values=2C AvroCollector<=3BGenericRecord>=3B col= lector=2C Reporter reporter) throws IOException {

=3B = =3B =3B =3B =3B =3B =3B =3B =3B =3B= =3B =3B =3B GenericRecord record =3D null=3B

&nbs= p=3B =3B =3B =3B =3B =3B =3B =3B =3B= =3B =3B =3B =3B for (GenericRecord value : values) {

=3B =3B =3B =3B =3B =3B =3B &nbs= p=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B -- code omitted here --

=3B =3B =3B &n= bsp=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B record =3D value=3B

<= div> =3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B &nb= sp=3B record.put("rowkey"=2C key)=3B =3B <=3B=3D=3D=3D this statemen= t causes problem

=3B =3B =3B =3B =3B &nbs= p=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B collector.collect(record)=3B

= =3B =3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B }

=3B =3B =3B &= nbsp=3B =3B =3B =3B}

As explained i= n my previous message=2C if I remove the statement record.put("rowkey"=2C k= ey)=2C the code works fine=2C in that the key values pairs passed to the ro= utine reduce() are correct. =3BBut if you add this statement=2C the ke= y values pairs passed to the routine reduce() are out of order=2C something= like (key1=2C values1)=2C (key2=2C values3) rather than (key2=2C values2).= =3BSome details are explained in my previous message. =3BIs &nbs= p=3Bthis problem relating to Hadoop binary iterators or Avro deserializatio= n code? =3BThanks.

Ey-Chih Chow

From: eychih@hotmail.com
To: user@avro.apache.o= rg
Subject: is this a bug?
Date: Wed=2C 2 Mar 2011 13:05:55 -0800
=
Hi=2C

I am working on an Avro MR job and encountering an= issue with AvroReducer<=3BUtf8=2C GenericRecord=2C GenericRecord>=3B. = The corresponding reduce() routine is implemented in the following way:

public void reduce(Utf8 key=2C Iterable<=3BGenericR= ecord>=3B values=2C AvroCollector<=3BGenericRecord>=3B collector=2C R= eporter reporter) throws IOException {

=3B =3B &n= bsp=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B.

=3B =3B =3B =3B =3B =3B =3B =3B &= nbsp=3B =3B =3B =3B =3B =3B =3B =3B &nbs= p=3B =3B.

=3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B =3B= =3B =3B =3B =3B.

=3B =3B &nb= sp=3B =3B GenericRecord record =3D null=3B

=3B&n= bsp=3B =3B =3B for (GenericRecord value : values) {

&nb= sp=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B =3B= =3B .

=3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B &nb= sp=3B =3B =3B =3B .

=3B =3B =3B &nbs= p=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B .

= =3B =3B =3B =3B =3B =3B =3Brecord =3D value=3B=

=3B =3B =3B =3B =3B =3B =3Brecord.put("rowkey"=2C k= ey)=3B

=3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B =3B= =3B =3B =3B =3B .

=3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B &nb= sp=3B =3B =3B =3B =3B =3B =3B .

&nb= sp=3B =3B =3B =3B =3B =3B =3B =3B = =3B =3B =3B =3B =3B =3B =3B =3B =3B= =3B .

=3B =3B =3B =3B =3B =3B = =3Bcollector.collect(record)=3B

=3B =3B =3B &n= bsp=3B =3B }

} =3B

If I comment out the= statement in red in the above code=2C the reduce function gets called prop= erly with CORRECT key values pairs passed to reduce(). =3BHowever=2C i= f I add the statement in red to the routine=2C the reduce function is calle= d with WRONG key values pairs=2C in the sense that key2 paired with values3= =2C instead of values2=2C when passed to the reduce() routine. =3BI tr= aced this problem by including Hadoop source code=2C such as ReduceTask.jav= a=2C Task.java=2C and Avro source code=2C such as HadoopReducer.java=2C Had= oopReducerBase.java=2C and all the serialization code. =3BThe problem = showed up on the second call of the reduce()=2C but I can not locate the ex= act place that cause the problem. =3BMy intuition is that this is incu= rred in either the hadoop iterators after merge sort or Avro deserializatio= n. =3BIs there anybody can help me on this? =3BThanks.

Ey-Chih Chow =3B =3B =3B

= --_1047860a-22eb-43a6-9c3d-436bcd2c8380_--