avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ey-chih chow <eyc...@hotmail.com>
Subject RE: map reduce job for avro 1.5 broken
Date Mon, 28 Mar 2011 22:20:14 GMT

We got another stack trace as follows:
java.lang.ArrayIndexOutOfBoundsException: 40	at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)	at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)	at org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86)
at org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68)
at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136)	at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242)	at org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47)
at com.ngmoco.ngpipes.sourcing.sessions.NgSessionReducer.reduce(NgSessionReducer.java:74)
at com.ngmoco.ngpipes.sourcing.sessions.NgSessionReducer.reduce(NgSessionReducer.java:1)	at
org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60)	at org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
at org.apache.hadoop.mapred.Child$4.run(Child.java:240)	at java.security.AccessController.doPrivileged(Native
Method)	at javax.security.auth.Subject.doAs(Subject.java:396)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234)
Ey-Chih Chow
From: eychih@hotmail.com
To: user@avro.apache.org
Subject: map reduce job for avro 1.5 broken
Date: Mon, 28 Mar 2011 14:45:23 -0700








Hi,
We have an avro map/reduce job used to be working with avro 1.4, but broken with avro 1.5
when the reducer tried to do de-serilization.  By looking at the trace, it looked like the
reducer was broken when trying to resolve 'union' of a 'map' definition in our avdl schema.
 We have three fields in our schema relating to this.  These are:
	union	{map <union {int,long,float,string}>, null} evpl;
	union 	{map <union {int,long,float,string}>, null} plst;
	union 	{map <union {int,long,float,string}>, null} change;

Can anybody let me know if this is a 1.5 bug?  The stack trace was as follows:
java.lang.ArrayIndexOutOfBoundsException: -1576799025	at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364)
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)	at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:232)	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:141)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:86)
at org.apache.avro.mapred.AvroSerialization$AvroWrapperDeserializer.deserialize(AvroSerialization.java:68)
at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136)	at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242)	at org.apache.avro.mapred.HadoopReducerBase$ReduceIterable.next(HadoopReducerBase.java:47)
at com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:46)	at com.ngmoco.ngpipes.etl.NgEventETLReducer.reduce(NgEventETLReducer.java:1)
at org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60)	at org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
at org.apache.hadoop.mapred.Child$4.run(Child.java:240)	at java.security.AccessController.doPrivileged(Native
Method)	at javax.security.auth.Subject.doAs(Subject.java:396)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234)
Thanks.
Ey-Chih Chow 		 	   		  
Mime
View raw message