avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Moiz Arafat <moiz.ara...@teamaol.com>
Subject Issue while Writing Output of PigUDF to AVRO File
Date Tue, 16 May 2017 18:42:23 GMT
Hi ,

I have a PigUDF which returns object of type Map<String, String[]> and type casted to
map[{value: (array: chararray)}] in the pig script . Then I use org.apache.pig.piggybank.storage.avro.AvroStorage
to write it to a file with schema provided as 

{"name":"Ekv","type":{"type":"map","values":{"type":"array","items":{"type":"string","avro.java.string":"String"}},"avro.java.string":"String"},"doc":"This
is the map of data. The id will be the key and the value will be an array of strings","default":{}},

But the script is failing with Error :

2017-05-16 17:14:11.464 ERROR 33212 --- [           main] org.apache.pig.tools.pigstats.PigStats
  : ERROR 0: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate
exception from backed error: Error: java.io.IOException: org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.RuntimeException: Unsupported type in array:class [Ljava.lang.String;
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:83)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:667)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.RuntimeException:
Unsupported type in array:class [Ljava.lang.String;
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263)
	at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49)
	at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:810)
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
	... 17 more
Caused by: java.lang.RuntimeException: Unsupported type in array:class [Ljava.lang.String;
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.getArraySize(PigAvroDatumWriter.java:397)
	at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
	at org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:173)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:69)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.writeRecord(PigAvroDatumWriter.java:365)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.writeUnion(PigAvroDatumWriter.java:113)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:82)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.writeRecord(PigAvroDatumWriter.java:365)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
	at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
	... 20 more

Has any one faced similar issue before ? 

Thanks,
Moiz
Mime
View raw message