Hi=2C

Current= ly I have a MR job needs to use my own Key class to support 2nd sort in the= MR job.

The originally job is using Avro String t= ype as the mapper output like this format:

pu= blic class MyMapper extends MapReduceBase implements Mapper<=3BLongWritab= le=2C Text=2C AvroKey<=3BCharSequence>=3B=2C

=3B = =3B =3B =3B AvroValue<=3BOneAvroSpecificRecordObject>=3B>= =3B

Right now=2C I need to change the key fr= om Text to a custom Key object=2C as I need to control complex sorting orde= r and support 2nd sort in my MR job.

So I create a= CustomKeyObject (PartitionKey class)=2C which contains 3 Long values and 4= String values. This key class implements =3BWritableComparable and I a= lso have my =3BKeyComparator and KeyGroupComparator class implementatio= n ready.

So in this case=2C I want to change my ma= pper for the new format:

public class MyMappe= r extends MapReduceBase implements Mapper<=3BLongWritable=2C Text=2C Avro= Key<=3B =3BCustomKeyObject =3B>=3B=2C

=3B = =3B =3B =3B AvroValue<=3BOneAvroSpecificRecordObject>=3B>= =3B

Here comes the problem=2C I don't know w= hat kind of schema I can use in my driver class for this key.

Originally=2C the driver will have following line:

AvroJob.setMapOutputSchema(conf=2C Pair.getPairSchema(Schema.c= reate(Schema.Type.STRING)=2C =3BOneAvroSpecificRecordObject.SCHEMA$))= =3B

So my question is what kind of schema I should= use above to replay the TYPE.STRING?

Here are som= e things I tried=2C and the error I got:

1) I trie= d with a Union Schema=2C with 3 Long Types and 4 String Types. It does NOT = work=2C as union cannot contain duplicate types.

2) Then I think = I need to create an anonymous record schema=2C it should work for my case. = So here is what I do:

=3B =3B First=2C in the code=2C a= dd the schema definition:

=3B =3B =3BString keySche= ma =3D "type........." // create a record schema with 3 long types and 4 st= ring types

=3B =3B Then=2C generate the schema at runti= me in my code:

=3B =3BAvroJob.setMapOutputSchema(conf= =2C Pair.getPairSchema(new Schema.Parser().parse(keySchema)=2C =3BOneAvroSpecificRecordObject.SCHEMA$)= )=3B

=3B =3BT= his works fine for all my mapper stage=2C but in the reducer part=2C it fai= led with the following error:

=3B =3Bjava.lang.ClassCastException: org.apache.avr= o.generic.GenericData$Record cannot be cast to PartitionKey

My re= ducer likes this:

myReducer implements Reducer<=3BAvroKey<=3B= =3BPartitionKey =3B>=3B=2C AvroValue<=3B =3BOneAvroSpecifi= cRecordObject =3B>=3B=2C NullWritable=2C NullWritable>=3B

It looks like if I use anonymous record schema=2C it will use genericData$= Record=2C which I cannot cast to PartitionKey class I want.