flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: Compression of network traffic
Date Mon, 24 Nov 2014 09:23:06 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

What reduction in network traffic can we expect from using compression?

- -s



On 11/24/2014 10:20 AM, Stephan Ewen wrote:
> Will the compression Codec will be inserted in the Netty loops, or
> before that?
> 
> In any case, would it make sense to prototype this on the current
> code and forward port this to the new network stack later? I assume
> the code would mostly be similar, especially all the JNI vs. Java
> considerations and tricks. Am 23.11.2014 23:11 schrieb "Ufuk
> Celebi" <uce@apache.org>:
> 
>> 
>> On 23 Nov 2014, at 21:04, Robert Metzger <rmetzger@apache.org>
>> wrote:
>> 
>>> Ufuk can probably elaborate on the details, but he told me some
>>> time ago that we can add code to compress outgoing network
>>> buffers using snappy or other fast compression algorithms. In
>>> particular I/O intensive
>> applications
>>> would benefit from such a change.
>> 
>> I agree and think that compressing shuffle data is generally a
>> good idea (given codecs like Snappy et al.)
>> 
>> I've opened a corresponding issue ( 
>> https://issues.apache.org/jira/browse/FLINK-1275). The network
>> layer is going through a set of big changes at the moment. I
>> think it makes sense to wait for them to be finished/merged,
>> before working on this.
>> 
>> @Viktor: If you want to work on this, feel free to comment on the
>> JIRA issue and we can discuss at which points you would have to
>> extend the system.
>> 
>> – Ufuk
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJUcvjyAAoJEFWNjCUTnssiCYgQAK5QzsYPW9Q+C8BUJBxfeudg
61zFPW3cfydfD4L6UXghwz/j7addMm5zdY/xJsFgmN4NAXrtacDfr6+KPyB9glZ0
E8E201q8NN6Czek/TtWEG0SSLj+KXqRAwJBHfBCVaCV/lD7wEAjef2sSFpVL5RK9
0RyS5UeCBW/RxTp/fGXWfy2d9G0SSTIy78/u8Y9nxflD6KnZVGZYZA7jrP/h8Wis
ry1ZWLPHKpEbaC+MNbYX1mMJaf9lH0liJ/HopCIUzrxfiwskub17NkfVC/0Uwwc3
r1+/pHFYNfO3FLIkXxWYRLgNyGW6LCE1zmM/BGUo1/C9/rtuiRl24io+G+4V/ac6
tZ9jxjwHQkWGFscoep2p7E8pGVezV1QqYv3prWXaZ2+TUKKzlbd6QCL8i2qmymeb
ABy88cpN1U4c/p6GDrva1+llyBG1PK07awAfFmK3OA7VK3MLW1pyHuJiLjOZaFP1
r2K9T5/d2YZa7nPFaJs8bLXsV/WMzFJDAOaILcaCDymxF1xnNu7iwAyuJhg/i6ht
KaOiM45JnqC++9MQ2D83zyt/+Yc/OT+06hVN7hdRaLFD+OqG7vNv7LZC+cfAWmFE
KyJb9kXNVDxhJXAWyOyKN1nzBtDfaugghGBzvX0u3QP/2uGX1Gca2oqCJvNesOM4
dBlCMt88/gkYjWXadjnK
=0q6A
-----END PGP SIGNATURE-----

Mime
View raw message