giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Re: Review Request 12377: Provide an option to do request compression
Date Wed, 10 Jul 2013 18:01:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12377/
-----------------------------------------------------------

(Updated July 10, 2013, 6:01 p.m.)


Review request for giraph.


Changes
-------

Nitay's comments


Bugs: GIRAPH-713
    https://issues.apache.org/jira/browse/GIRAPH-713


Repository: giraph-git


Description
-------

In some cases, network is much slower than all the computation stuff we do, and we could benefit
from compressing requests.

I am using LZ4 compression. Tried out two versions of Snappy and some java ones, and LZ4 had
the least CPU overhead, and about the same speed as Snappy. It's also easy to plug in your
own compressor. In cases when we are not bounded by network, CPU overhead of using compression
is about 10% (testing with PageRankBenchmark) and time stays about the same. Depending on
how slow your network is and how good your data compresses this can lead to big time savings.
Also added compression to metrics, from one of the pagerank iterations:
https://gist.github.com/majakabiljo/5962269


Diffs (updated)
-----

  giraph-core/pom.xml cab0157 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/LZ4RequestCodec.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/NoOpRequestCodec.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestCodec.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java 6eb6549

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java 83b408e

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 6655834 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f 
  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
ed63192 
  giraph-core/src/main/java/org/apache/giraph/metrics/GiraphMetrics.java 7d980ea 
  giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac 
  giraph-core/src/main/java/org/apache/giraph/utils/Sizes.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java 20ed92b

  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java 9ff1242

  giraph-core/src/test/java/org/apache/giraph/comm/LZ4RequestCodecTest.java PRE-CREATION 
  pom.xml 72fe6f2 

Diff: https://reviews.apache.org/r/12377/diff/


Testing
-------

mvn clean verify
PageRank on cluster
Added test for compressor
Verified that NoOpCompressor has the same performance as before


Thanks,

Maja Kabiljo


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message