flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Klemke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8186) AvroInputFormat regression: fails to deserialize GenericRecords on standalone cluster with hadoop27 compat
Date Fri, 01 Dec 2017 16:44:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274609#comment-16274609
] 

Sebastian Klemke commented on FLINK-8186:
-----------------------------------------

Regarding binaries: Yes, this is the runtime I'm using. I'm compiling against 1.4 snapshot
via maven. And for the avro files: I'll test with https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/test/resources/testdata.avro

> AvroInputFormat regression: fails to deserialize GenericRecords on standalone cluster
with hadoop27 compat
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-8186
>                 URL: https://issues.apache.org/jira/browse/FLINK-8186
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 1.4.0
>            Reporter: Sebastian Klemke
>            Priority: Minor
>         Attachments: GenericRecordCount.java, pom.xml
>
>
> The following job runs fine on a Flink 1.3.2 cluster, but fails on a Flink 1.4.0 RC2
standalone cluster, "hadoop27" flavour:
> {code}
> public class GenericRecordCount {
>     public static void main(String[] args) throws Exception {
>         String input = ParameterTool.fromArgs(args).getRequired("input");
>         ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
>         long count = env.readFile(new AvroInputFormat<>(new Path(input), GenericRecord.class),
input)
>                 .count();
>         System.out.printf("Counted %d records\n", count);
>     }
> }
> {code}
> Runs fine in LocalExecutionEnvironment and also on no-hadoop flavour standalone cluster,
though. Exception thrown in Flink 1.4.0 hadoop27:
> {code}
> 12/01/2017 13:22:09     DataSource (at readFile(ExecutionEnvironment.java:514) (org.apache.flink.formats.avro.AvroInputFormat))(4/4)
switched to FAILED
> java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.avro.generic.GenericRecord.<init>()
>         at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353)
>         at org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369)
>         at org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901)
>         at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
>         at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
>         at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
>         at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
>         at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
>         at org.apache.flink.formats.avro.AvroInputFormat.nextRecord(AvroInputFormat.java:165)
>         at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:167)
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoSuchMethodException: org.apache.avro.generic.GenericRecord.<init>()
>         at java.lang.Class.getConstructor0(Class.java:3082)
>         at java.lang.Class.getDeclaredConstructor(Class.java:2178)
>         at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
>         ... 11 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message