avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Barefoot Burns <bpburn...@gmail.com>
Subject Convert GenericRecord objects between schemas
Date Wed, 31 May 2017 07:48:54 GMT

I am trying to multiplex GenericRecord objects to a set of DataFileWriter
objects. Each DataFileWriter expects its own schema, and the GenericRecord
objects are written with a "superset" schema that is compatible with the
schemas assigned to the DataFileWriters. Is there a way to convert
individual GenericRecord objects between schemas without re-reading them
from their source (in this case a DataFileStream)? If I simply call
append() on a DataFileWriter with a GenericRecord, I get an exception
complaining that the schema of the GenericRecord does not match the
DataFileWriter. If I set the reader schema in the DataFileStream to the
schema expected by a given DataFileWriter, I do not get these errors, but I
would prefer to not have to open the stream multiple times to write out the
records to the different writers. I feel like there should be some obvious
tool that leverages schema evolution that could do this, but I cannot find
it if it exists.

Thank you!

View raw message