giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-713) Provide an option to do request compression
Date Thu, 03 Jul 2014 23:45:34 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052009#comment-14052009
] 

Pavan Kumar commented on GIRAPH-713:
------------------------------------

I created a patch for netty upgrade on the dependent task
(note that the additional changes are because we do not want to use cache - some internal
testing showed high perf regression with caching turned on in netty)

ratio of sent bytes with different compression algos for pagerank run on a graph with 1million
vertices and 85 million edges

none vs jdkzlib = 2.5
none vs snappy = 1.5

however in my runs, the overhead of compression is much  more than saving obtained. this is
not a comprehensive experiment, though.
our network is much faster, etc. so might be useful for others, in anycase, the change is
non-intrusive, by default compression is turned off, so no regression. Users, can turn on
compression by providing the option 
-Dgiraph.nettyCompressionAlgorithm=SNAPPY
or 
-Dgiraph.nettyCompressionAlgorithm=INFLATE

> Provide an option to do request compression
> -------------------------------------------
>
>                 Key: GIRAPH-713
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-713
>             Project: Giraph
>          Issue Type: New Feature
>            Reporter: Maja Kabiljo
>         Attachments: GIRAPH-713.patch, GIRAPH-713_1.patch
>
>
> In some cases, network is much slower than all the computation stuff we do, and we could
benefit from compressing requests.
> https://reviews.apache.org/r/12377/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message