hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AnilKumar B <akumarb2...@gmail.com>
Subject Re: Facing problem while emitting GenericData$Record
Date Sun, 09 Feb 2014 11:26:47 GMT
Thanks once again Harsh.

Thanks & Regards,
B Anil Kumar.


On Sat, Feb 8, 2014 at 11:48 PM, Harsh J <harsh@cloudera.com> wrote:

> I've responded to this on your original post over the appropriate
> user@avro lists: http://search-hadoop.com/m/icC8CkA4h1
>
> On Sat, Feb 8, 2014 at 11:47 PM, AnilKumar B <akumarb2010@gmail.com>
> wrote:
> > Hi,
> >
> > In my Mapreduce job, I am using AvroKeyValueOutputFormat as one of my
> > MultipleOutputs. So I declared my multiple outputs as below.
> >
> >  MultipleOutputs.addNamedOutput(stage2Job,
> >         SessionConstants.COMPLETED_SESSIONS,
> >         AvroKeyValueOutputFormat.class, AvroKey.class, AvroValue.class);
> >
> > And In reducer, I am constructing and emitting GenericData.Record for the
> > below schema:
> >
> > sessionSchema:
> > {"namespace": "ci.avro",
> >  "type": "record",
> >  "name": "Session",
> >  "fields": [
> >    {"name":"Common", "type": {
> >            "type": "map", "values":"string"}},
> >    {"name":"events",
> >     "type": {
> >         "type": "array",
> >      "items":{
> >      "name":"Event",
> >      "type":"map",
> >      "values":"string"}
> >      }
> >     }
> >  ]
> > }
> >
> > eventSchema:
> > {"namespace": "ci.avro",
> >  "type": "record",
> >  "name": "AvroEvent",
> >  "fields": [
> >     {"name":"Event",
> >       "type": {
> >            "type": "map", "values":"string"
> >               }
> >     }
> >  ]
> > }
> >
> >
> > //
> > sessionRecord = new AvroValue<GenericData.Record>(null);
> >
> > //record generation
> >
> >  GenericData.Record record = new GenericData.Record(sessionSchema);
> >     GenericData.Record eRecord = new GenericData.Record(eventSchema);
> >     GenericData.Array<GenericData.Record> eventRecords =
> >         new GenericData.Array<GenericData.Record>(vc.getEvents().size(),
> >             sessionSchema.getField("events").schema());
> >     record.put("Common", vc.getCommon().getM_parameterMap());
> >     for (Event ev : vc.getEvents()) {
> >       eRecord = new GenericData.Record(eventSchema);
> >       eRecord.put("Event", ev.getM_parameterMap());
> >       eventRecords.add(eRecord);
> >     }
> >     record.put("events", eventRecords);
> >
> >     sessionRecord.datum(record);
> >
> >
> > // record emmitted as below
> > context.getConfiguration().set(CONF_OUTPUT_KEY_SCHEMA,
> >           Schema.create(Schema.Type.STRING).toString());
> >       context.getConfiguration().set(CONF_OUTPUT_VALUE_SCHEMA,
> >           sessionSchema.toString());
> >       multipleOutputs.write(SessionConstants.COMPLETED_SESSIONS,
> >           new AvroKey<String>(key.toString()), sessionRecord,
> >           SessionConstants.COMPLETED_SESSIONS);
> >
> > But I am getting below exception. I am also declaring
> > "avro.schema.output.value" as sessionSchema.toString().  What could be
> the
> > issue?
> >
> > I am not understanding why it's trying to cast GenericData$Record to
> > java.util.Map.
> >
> > Exception:
> > org.apache.avro.file.DataFileWriter$AppendWriteException:
> > java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record
> > cannot be cast to java.util.Map
> >         at
> > org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
> >         at
> >
> org.apache.avro.mapreduce.AvroKeyValueRecordWriter.write(AvroKeyValueRecordWriter.java:127)
> >         at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs$RecordWriterWithCounter.write(MultipleOutputs.java:304)
> >         at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> >         at
> >
> com.paypal.ci.CISuperSessionStage2AvroReducer.reduce(CISuperSessionStage2AvroReducer.java:156)
> >         at
> >
> com.paypal.ci.CISuperSessionStage2AvroReducer.reduce(CISuperSessionStage2AvroReducer.java:24)
> >         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> >         at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> >         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> >         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at javax.security.auth.Subject.doAs(Subject.java:396)
> >         at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> >         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > Caused by: java.lang.ClassCastException:
> > org.apache.avro.generic.GenericData$Record cannot be cast to
> java.util.Map
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.getMapSize(GenericDatumWriter.java:194)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:173)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:69)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:138)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.writeArray(ReflectDatumWriter.java:64)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> >         at
> >
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> >         at
> >
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> >         at
> > org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
> >
> >
> >
> > Thanks & Regards,
> > B Anil Kumar.
> > Thanks & Regards,
> > B Anil Kumar.
>
>
>
> --
> Harsh J
>

Mime
View raw message