beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Created] (BEAM-2677) without specifying a schema
Date Mon, 24 Jul 2017 22:08:00 GMT
Eugene Kirpichov created BEAM-2677:

             Summary: without specifying a schema
                 Key: BEAM-2677
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-core
            Reporter: Eugene Kirpichov
            Assignee: Eugene Kirpichov

Sometimes it is inconvenient to require the user of to specify a Schema
for the Avro files they are reading, especially if different files may have different schemas.

It is possible to read GenericRecord objects from an Avro file, however it is not possible
to provide a Coder for GenericRecord without knowing the schema: a GenericRecord knows its
schema so we can encode it into a byte array, but we can not decode it from a byte array without
knowing the schema (and encoding the full schema together with every record would be impractical).

Instead, a reasonable approach is to treat schemaless GenericRecord as unencodable and use
the same approach as JdbcIO - a user-specified parse callback.

Suggested API: AvroIO.parseGenericRecords(SerializableFunction<GenericRecord, T> parseFn).from(filepattern).

CC: [~mkhadikov] [~reuvenlax]

This message was sent by Atlassian JIRA

View raw message