pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nandor Kollar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5056) Fix AvroStorage writing enums
Date Wed, 09 Nov 2016 13:58:58 GMT

    [ https://issues.apache.org/jira/browse/PIG-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651025#comment-15651025
] 

Nandor Kollar commented on PIG-5056:
------------------------------------

Looks good to me, doesn't break Avro 1.7.5, and fixes problem with 1.8.1.

> Fix AvroStorage writing enums
> -----------------------------
>
>                 Key: PIG-5056
>                 URL: https://issues.apache.org/jira/browse/PIG-5056
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.16.0
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>              Labels: avro
>             Fix For: 0.17.0
>
>         Attachments: PIG-5056.patch
>
>
> Issue is observable with latest (1.8.1) Avro since it has an extra check for enum types
that the current 1.7.5 does not care about (see here: https://github.com/apache/avro/blob/release-1.8.1/lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumWriter.java#L163)
> This results in TestAvroStorage#testLoadRecordsWithEnums failing:
> {code}
> java.lang.Exception: java.io.IOException: org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.AvroTypeException: Not an enum: GOOD
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> Caused by: java.io.IOException: org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.AvroTypeException: Not an enum: GOOD
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:83)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:655)
> 	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> 	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> 	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException:
Not an enum: GOOD
> 	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> 	at org.apache.pig.impl.util.avro.AvroRecordWriter.write(AvroRecordWriter.java:115)
> 	at org.apache.pig.impl.util.avro.AvroRecordWriter.write(AvroRecordWriter.java:51)
> 	at org.apache.pig.builtin.AvroStorage.putNext(AvroStorage.java:520)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
> 	... 18 more
> Caused by: org.apache.avro.AvroTypeException: Not an enum: GOOD
> 	at org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:164)
> 	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:106)
> 	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> 	at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> 	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> 	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> 	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> 	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> 	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> 	... 22 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message