beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Florian Scharinger (JIRA)" <>
Subject [jira] [Commented] (BEAM-55) Allow users to compress FileBasedSink output files
Date Mon, 03 Oct 2016 01:07:21 GMT


Florian Scharinger commented on BEAM-55:

Hi Daniel,

At the time when I raised this, I was under the impression that codecs like Snappy are not
supported. We have changed our system design significantly so that we do not need to read
uncompressed Avro files from GCS. Having said that, we will need to produce compressed text
files for another part of our system in the future, so Jeffrey's contribution will be very


> Allow users to compress FileBasedSink output files
> --------------------------------------------------
>                 Key: BEAM-55
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Priority: Minor
> FileBasedSink (also TextIO.Write, AvroIO.Write, etc). does not have an option for compressing
its output.
> In general, we discourage compression because it limits or blocks scalably reading from
a file in parallel. However, users may want it -- so we should support the option (with appropriate

This message was sent by Atlassian JIRA

View raw message