flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Compression - AvroOutputFormat and over network ?
Date Mon, 18 Apr 2016 18:05:40 GMT
Hi Tarandeep,

I think for that you would need to set a codec factory on the DataFileWriter.
Sadly we don't expose that method to the user.

If you want, you can contribute this change to Flink. Otherwise, I can
quickly fix it.

Regards,
Robert


On Mon, Apr 18, 2016 at 2:36 PM, Ufuk Celebi <uce@apache.org> wrote:

> Hey Tarandeep,
>
> regarding the network part: not possible at the moment. It's pretty
> straight forward to add support for it, but no one ever got around to
> actually implementing it. If you would like to contribute, I am happy
> to give some hints about which parts of the system would need to be
> modified.
>
> – Ufuk
>
>
> On Mon, Apr 18, 2016 at 12:56 PM, Tarandeep Singh <tarandeep@gmail.com>
> wrote:
> > Hi,
> >
> > How can I set compression for AvroOutputFormat when writing files on
> HDFS?
> > Also, can we set compression for intermediate data that is sent over
> network
> > (from map to reduce phase) ?
> >
> > Thanks,
> > Tarandeep
>

Mime
View raw message