flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: Avro Parquet/Flink/Beam
Date Mon, 12 Dec 2016 16:25:18 GMT

Beam provides a AvroCoder/AvroIO that you can use, but not yet a 
ParquetIO (I created a Jira about that and started to work on it).

You can use the Avro reader to populate the PCollection and then use a 
custom DoFn to create the Parquet (waiting for the ParquetIO).


On 12/12/2016 05:19 PM, Newport, Billy wrote:
> Are there any examples showing the use of beam with avro/parquet and a
> flink runner? I see an avro reader for beam, is it a matter of writing
> another one for avro-parquet or does this need to use the flink
> HadoopOutputFormat for example?
> Thanks
> Billy

Jean-Baptiste Onofré
Talend - http://www.talend.com

View raw message