flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Compression of network traffic
Date Tue, 25 Nov 2014 19:53:50 GMT
I would start with a simple compression of network buffers as a blob.

At some point, Flink's internal data layout may become columnar, which
should also help the blob-style compression, because more similar strings
will be within one window...

On Tue, Nov 25, 2014 at 11:26 AM, Viktor Rosenfeld <
viktor.rosenfeld@tu-berlin.de> wrote:

> Hi,
> A codec like Snappy would work on an entire network buffer as one big blob,
> right? I was more thinking along the lines of compressing individual tuples
> fields by treating them as columns, e.g., using frame-of-reference encoding
> and bit backing. Compression on tuple fields should yield much better
> results than compressing the entire blob. Given that Flink controls the
> serialization process this should be transparent to other layers in the
> code. Not sure it is worth the effort though.
> Cheers,
> Viktor
> --
> View this message in context:
> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Compression-of-network-traffic-tp2568p2607.html
> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
> archive at Nabble.com.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message