I would like to use MR2 in conjunction with avro but cannot find too m= uch documentation on the topic. Do you have any pointers in that region ?

AVRO 1.7.1 does not have any AvroReducer / Mapper in the mapreduce pac= kage. I didnt look into it enough to see if perhaps the compatibility with = the v2 is solved under the hood transparently now.

In short I am having tremendous trouble finding documentation on the t= opic. =3B

Hopefully you guys are able to help me along.

2012/9/12 Frank Kootte <= =3B frankkootte@g= mail.com>=3B

Very interesting concept you mention there - avro projections !
This sounds indeed like a clever way to leverage the avro capability o= f comparance without deserialisation which will be obviously beneficial.
Now as with a lot of avro related hadoop topics I am not able to find = a clear example but from what I did mention to find I would like to get you= r feedback on my question -

Does avro projection involve defining a secondary schema describing on= ly the desired subset of fields ?

Does this then imply that when I define my own AvroKeyComparator<=3B= A>=3B the byte arrays will only contain the data for set A ? =3B
How should the BinaryCompare be used differently from the base impl in= =3BAvroKeyComparator ?

Secondary I've tried to implement a custom AvroKeyComparator and in sp= ecific the - =3Bcompare(byte[] b1=2C int= s1=2C int l1=2C byte[] b2=2C int s2=2C int l2) =3B - method. =3B

I am wowfully unaware on how to exactly= do this and cannot find a lot of examples on the topic. =3B

Could you write me a small sample of ps= eudo code perhaps ?

Or point me to some documentation to ge= t me on my way ?

2012/9/12 Jacob Metcalf <= =3Bjacob_met= calf@hotmail.com>=3B

Frank

I have spent a bit of time doing this recently but with MR2 and CDH4 w= hich may not be appropriate to your use case. =3BHowever assuming some similarities=2C =3BI suspect your problem is that you also need to override =3Bcompare= (byte[] b1=2C int s1=2C int l1=2C byte[] b2=2C int s2=2C int l2) on AvroKey= Comparator. =3B

The advantage to Avro is that Hadoop do= es not need to deserialize to sort in the shuffle. This function in RawComp= arator allows Hadoop to quickly compare the bytes directly.

Whilst this seems a bit daunting my trick to= doing this in MR2 is to leverage Avro's excellent support for projections = - subsets of schemas. For example let's say you want to "=3Bgroup"= =3B by attribute A but then "=3Bsort"=3B by attribute B. In this case I would use a composite key with schema {A=2C B} and the out = of the box =3BAvroKeyComparator as the sort comparator. Then I w= ould implement my own grouping comparator which uses a schema of just {A} t= hen uses the BinaryData function to compare:

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.avro/avro/= 1.4.0/org/apache/avro/mapred/AvroKeyComparator.java

I assume you can do something similar in MR1.

Regards

Jacob

>=3B Subject: Secondary sort in hadoop with avro
>=3B From: kotesk= ie@gmail.com
>=3B Date: Tue=2C 11 Sep 2012 17:36:06 +=3B0200
>=3B To: user@a= vro.apache.org

>=3B
>=3B I need to implement secondary sort within an avro based MR sequence.= I however find little to documentation or examples online.
>=3B I would like to implement this by overriding the 'int compare(AvroWr= apper<=3BT>=3B x=2C AvroWrapper<=3BT>=3B y)' method but I fail to h= ave it invoked.
>=3B Does anybody have experience implementing secondary sort on deserial= ised avro objects ?
>=3B
>=3B Some help=2C advise or pointers will be very much appreciated !

--
Mvrgr. Frank =3B

--
Mvrgr. Frank =3B