Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5426B1097D for ; Sun, 9 Feb 2014 11:27:28 +0000 (UTC) Received: (qmail 63707 invoked by uid 500); 9 Feb 2014 11:27:20 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 62861 invoked by uid 500); 9 Feb 2014 11:27:13 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 62843 invoked by uid 99); 9 Feb 2014 11:27:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Feb 2014 11:27:12 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of akumarb2010@gmail.com designates 209.85.216.169 as permitted sender) Received: from [209.85.216.169] (HELO mail-qc0-f169.google.com) (209.85.216.169) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Feb 2014 11:27:08 +0000 Received: by mail-qc0-f169.google.com with SMTP id w7so8915642qcr.14 for ; Sun, 09 Feb 2014 03:26:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=7EzpPxhPeVwsyuA2befWWhm1l5XEFMs9zuzX/0E0S50=; b=es8pO4GmD5fQn1w6GWYjz9Tk4XZexKNz+0yPP94FVhkTgNcFPUNLJopZO+b3PD2KSb mgNqh+HuGaN3+goA5oQ5pP9yDo9v/C4sGpK0Viq5rqT4tN7/nj1M/ACleyoYEvEYP391 n0APVsD3xx0L8EmlDZRPYm9SersdUz3KAjvh6GzDa1LD3IChLuImhwzakxzwauEg0Lk6 NLDBYonOga1FP+jy1X9/k0J2RNZSUKkF9qqmsqXPIXchK1zCGBe9pWWBr9deg5Za3NLQ i5cJOUCaoC4r8v9Ux38co1/q2KT1kjLdq3LaRp6OJugPrAlvxRChOenaiQ22SJNwc0td 0cEA== MIME-Version: 1.0 X-Received: by 10.224.4.5 with SMTP id 5mr38296146qap.61.1391945207659; Sun, 09 Feb 2014 03:26:47 -0800 (PST) Received: by 10.96.193.3 with HTTP; Sun, 9 Feb 2014 03:26:47 -0800 (PST) In-Reply-To: References: Date: Sun, 9 Feb 2014 06:26:47 -0500 Message-ID: Subject: Re: Facing problem while emitting GenericData$Record From: AnilKumar B To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c219eaa642af04f1f77f57 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c219eaa642af04f1f77f57 Content-Type: text/plain; charset=ISO-8859-1 Thanks once again Harsh. Thanks & Regards, B Anil Kumar. On Sat, Feb 8, 2014 at 11:48 PM, Harsh J wrote: > I've responded to this on your original post over the appropriate > user@avro lists: http://search-hadoop.com/m/icC8CkA4h1 > > On Sat, Feb 8, 2014 at 11:47 PM, AnilKumar B > wrote: > > Hi, > > > > In my Mapreduce job, I am using AvroKeyValueOutputFormat as one of my > > MultipleOutputs. So I declared my multiple outputs as below. > > > > MultipleOutputs.addNamedOutput(stage2Job, > > SessionConstants.COMPLETED_SESSIONS, > > AvroKeyValueOutputFormat.class, AvroKey.class, AvroValue.class); > > > > And In reducer, I am constructing and emitting GenericData.Record for the > > below schema: > > > > sessionSchema: > > {"namespace": "ci.avro", > > "type": "record", > > "name": "Session", > > "fields": [ > > {"name":"Common", "type": { > > "type": "map", "values":"string"}}, > > {"name":"events", > > "type": { > > "type": "array", > > "items":{ > > "name":"Event", > > "type":"map", > > "values":"string"} > > } > > } > > ] > > } > > > > eventSchema: > > {"namespace": "ci.avro", > > "type": "record", > > "name": "AvroEvent", > > "fields": [ > > {"name":"Event", > > "type": { > > "type": "map", "values":"string" > > } > > } > > ] > > } > > > > > > // > > sessionRecord = new AvroValue(null); > > > > //record generation > > > > GenericData.Record record = new GenericData.Record(sessionSchema); > > GenericData.Record eRecord = new GenericData.Record(eventSchema); > > GenericData.Array eventRecords = > > new GenericData.Array(vc.getEvents().size(), > > sessionSchema.getField("events").schema()); > > record.put("Common", vc.getCommon().getM_parameterMap()); > > for (Event ev : vc.getEvents()) { > > eRecord = new GenericData.Record(eventSchema); > > eRecord.put("Event", ev.getM_parameterMap()); > > eventRecords.add(eRecord); > > } > > record.put("events", eventRecords); > > > > sessionRecord.datum(record); > > > > > > // record emmitted as below > > context.getConfiguration().set(CONF_OUTPUT_KEY_SCHEMA, > > Schema.create(Schema.Type.STRING).toString()); > > context.getConfiguration().set(CONF_OUTPUT_VALUE_SCHEMA, > > sessionSchema.toString()); > > multipleOutputs.write(SessionConstants.COMPLETED_SESSIONS, > > new AvroKey(key.toString()), sessionRecord, > > SessionConstants.COMPLETED_SESSIONS); > > > > But I am getting below exception. I am also declaring > > "avro.schema.output.value" as sessionSchema.toString(). What could be > the > > issue? > > > > I am not understanding why it's trying to cast GenericData$Record to > > java.util.Map. > > > > Exception: > > org.apache.avro.file.DataFileWriter$AppendWriteException: > > java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record > > cannot be cast to java.util.Map > > at > > org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296) > > at > > > org.apache.avro.mapreduce.AvroKeyValueRecordWriter.write(AvroKeyValueRecordWriter.java:127) > > at > > > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs$RecordWriterWithCounter.write(MultipleOutputs.java:304) > > at > > > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370) > > at > > > com.paypal.ci.CISuperSessionStage2AvroReducer.reduce(CISuperSessionStage2AvroReducer.java:156) > > at > > > com.paypal.ci.CISuperSessionStage2AvroReducer.reduce(CISuperSessionStage2AvroReducer.java:24) > > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177) > > at > > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) > > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > Caused by: java.lang.ClassCastException: > > org.apache.avro.generic.GenericData$Record cannot be cast to > java.util.Map > > at > > > org.apache.avro.generic.GenericDatumWriter.getMapSize(GenericDatumWriter.java:194) > > at > > > org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:173) > > at > > > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:69) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143) > > at > > > org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:138) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.writeArray(ReflectDatumWriter.java:64) > > at > > > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143) > > at > > > org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175) > > at > > > org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104) > > at > > > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143) > > at > > > org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175) > > at > > > org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104) > > at > > > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66) > > at > > > org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143) > > at > > > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58) > > at > > org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290) > > > > > > > > Thanks & Regards, > > B Anil Kumar. > > Thanks & Regards, > > B Anil Kumar. > > > > -- > Harsh J > --001a11c219eaa642af04f1f77f57 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks once again Harsh.

Thanks & Regards,
B Anil Kumar.


On Sat, Feb 8, 2014 at 11:48 PM, Harsh J= <harsh@cloudera.com> wrote:
I've responded to this on your original post over the appropriate
user@avro lists: http://search-hadoop.com/m/icC8CkA4h1

On Sat, Feb 8, 2014 at 11:47 PM, AnilKumar B <akumarb2010@gmail.com> wrote:
> Hi,
>
> In my Mapreduce job, I am using AvroKeyValueOutputFormat as one of my<= br> > MultipleOutputs. So I declared my multiple outputs as below.
>
> =A0MultipleOutputs.addNamedOutput(stage2Job,
> =A0 =A0 =A0 =A0 SessionConstants.COMPLETED_SESSIONS,
> =A0 =A0 =A0 =A0 AvroKeyValueOutputFormat.class, AvroKey.class, AvroVal= ue.class);
>
> And In reducer, I am constructing and emitting GenericData.Record for = the
> below schema:
>
> sessionSchema:
> {"namespace": "ci.avro",
> =A0"type": "record",
> =A0"name": "Session",
> =A0"fields": [
> =A0 =A0{"name":"Common", "type": {
> =A0 =A0 =A0 =A0 =A0 =A0"type": "map", "values= ":"string"}},
> =A0 =A0{"name":"events",
> =A0 =A0 "type": {
> =A0 =A0 =A0 =A0 "type": "array",
> =A0 =A0 =A0"items":{
> =A0 =A0 =A0"name":"Event",
> =A0 =A0 =A0"type":"map",
> =A0 =A0 =A0"values":"string"}
> =A0 =A0 =A0}
> =A0 =A0 }
> =A0]
> }
>
> eventSchema:
> {"namespace": "ci.avro",
> =A0"type": "record",
> =A0"name": "AvroEvent",
> =A0"fields": [
> =A0 =A0 {"name":"Event",
> =A0 =A0 =A0 "type": {
> =A0 =A0 =A0 =A0 =A0 =A0"type": "map", "values= ":"string"
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 }
> =A0]
> }
>
>
> //
> sessionRecord =3D new AvroValue<GenericData.Record>(null);
>
> //record generation
>
> =A0GenericData.Record record =3D new GenericData.Record(sessionSchema)= ;
> =A0 =A0 GenericData.Record eRecord =3D new GenericData.Record(eventSch= ema);
> =A0 =A0 GenericData.Array<GenericData.Record> eventRecords =3D > =A0 =A0 =A0 =A0 new GenericData.Array<GenericData.Record>(vc.get= Events().size(),
> =A0 =A0 =A0 =A0 =A0 =A0 sessionSchema.getField("events").sch= ema());
> =A0 =A0 record.put("Common", vc.getCommon().getM_parameterMa= p());
> =A0 =A0 for (Event ev : vc.getEvents()) {
> =A0 =A0 =A0 eRecord =3D new GenericData.Record(eventSchema);
> =A0 =A0 =A0 eRecord.put("Event", ev.getM_parameterMap()); > =A0 =A0 =A0 eventRecords.add(eRecord);
> =A0 =A0 }
> =A0 =A0 record.put("events", eventRecords);
>
> =A0 =A0 sessionRecord.datum(record);
>
>
> // record emmitted as below
> context.getConfiguration().set(CONF_OUTPUT_KEY_SCHEMA,
> =A0 =A0 =A0 =A0 =A0 Schema.create(Schema.Type.STRING).toString());
> =A0 =A0 =A0 context.getConfiguration().set(CONF_OUTPUT_VALUE_SCHEMA, > =A0 =A0 =A0 =A0 =A0 sessionSchema.toString());
> =A0 =A0 =A0 multipleOutputs.write(SessionConstants.COMPLETED_SESSIONS,=
> =A0 =A0 =A0 =A0 =A0 new AvroKey<String>(key.toString()), session= Record,
> =A0 =A0 =A0 =A0 =A0 SessionConstants.COMPLETED_SESSIONS);
>
> But I am getting below exception. I am also declaring
> "avro.schema.output.value" as sessionSchema.toString(). =A0W= hat could be the
> issue?
>
> I am not understanding why it's trying to cast GenericData$Record = to
> java.util.Map.
>
> Exception:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.ClassCastException: org.apache.avro.generic.GenericData$Reco= rd
> cannot be cast to java.util.Map
> =A0 =A0 =A0 =A0 at
> org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296) > =A0 =A0 =A0 =A0 at
> org.apache.avro.mapreduce.AvroKeyValueRecordWriter.write(AvroKeyValueR= ecordWriter.java:127)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs$RecordWriterWit= hCounter.write(MultipleOutputs.java:304)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleO= utputs.java:370)
> =A0 =A0 =A0 =A0 at
> com.paypal.ci.CISuperSessionStage2AvroReducer.reduce(CISuperSessionSta= ge2AvroReducer.java:156)
> =A0 =A0 =A0 =A0 at
> com.paypal.ci.CISuperSessionStage2AvroReducer.reduce(CISuperSessionSta= ge2AvroReducer.java:24)
> =A0 =A0 =A0 =A0 at org.apache.hadoop.mapreduce.Reducer.run(Reducer.jav= a:177)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)=
> =A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.= java:418)
> =A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.Child$4.run(Child.java:255= )
> =A0 =A0 =A0 =A0 at java.security.AccessController.doPrivileged(Native = Method)
> =A0 =A0 =A0 =A0 at javax.security.auth.Subject.doAs(Subject.java:396)<= br> > =A0 =A0 =A0 =A0 at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1232)
> =A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.Child.main(Child.java:249)=
> Caused by: java.lang.ClassCastException:
> org.apache.avro.generic.GenericData$Record cannot be cast to java.util= .Map
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.getMapSize(GenericDatumWrit= er.java:194)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter= .java:173)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.ja= va:69)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.ja= va:143)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWrit= er.java:138)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.writeArray(ReflectDatumWrit= er.java:64)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.ja= va:68)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.ja= va:143)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWrit= er.java:114)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWrit= er.java:175)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWri= ter.java:104)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.ja= va:66)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.ja= va:143)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWrit= er.java:114)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWrit= er.java:175)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWri= ter.java:104)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.ja= va:66)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.ja= va:143)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.ja= va:58)
> =A0 =A0 =A0 =A0 at
> org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290) >
>
>
> Thanks & Regards,
> B Anil Kumar.
> Thanks & Regards,
> B Anil Kumar.



--
Harsh J

--001a11c219eaa642af04f1f77f57--