Mailing-List: contact giraph-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: giraph-dev@incubator.apache.org
Date: Wed, 9 May 2012 09:41:56 +0000 (UTC)
From: "Avery Ching (JIRA)" <jira@apache.org>
To: giraph-dev@incubator.apache.org
Message-ID: 
 <1226329728.43390.1336556516944.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <645141762.38119.1316254268777.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271245#comment-13271245 ] 

Avery Ching commented on GIRAPH-37:
-----------------------------------

Thanks Claudio.

Here are more results with a scaled up 10 worker setup:

Hadoop RPC:
hadoop jar ~/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=false -w 10 -V 10000000 -s 5 -e 2 -v
12/05/09 02:32:05 INFO mapred.JobClient:   Giraph Timers
12/05/09 02:32:05 INFO mapred.JobClient:     Total (milliseconds)=149880
12/05/09 02:32:05 INFO mapred.JobClient:     Superstep 3 (milliseconds)=21575
12/05/09 02:32:05 INFO mapred.JobClient:     Setup (milliseconds)=7428
12/05/09 02:32:05 INFO mapred.JobClient:     Shutdown (milliseconds)=174
12/05/09 02:32:05 INFO mapred.JobClient:     Vertex input superstep (milliseconds)=39558
12/05/09 02:32:05 INFO mapred.JobClient:     Superstep 0 (milliseconds)=16887
12/05/09 02:32:05 INFO mapred.JobClient:     Superstep 4 (milliseconds)=18613
12/05/09 02:32:05 INFO mapred.JobClient:     Superstep 5 (milliseconds)=3292
12/05/09 02:32:05 INFO mapred.JobClient:     Superstep 2 (milliseconds)=21313
12/05/09 02:32:05 INFO mapred.JobClient:     Superstep 1 (milliseconds)=21035

Netty:
hadoop jar ~/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -w 10 -V 10000000 -s 5 -e 2 -v
12/05/09 02:35:06 INFO mapred.JobClient:   Giraph Timers
12/05/09 02:35:06 INFO mapred.JobClient:     Total (milliseconds)=59270
12/05/09 02:35:06 INFO mapred.JobClient:     Superstep 3 (milliseconds)=11827
12/05/09 02:35:06 INFO mapred.JobClient:     Setup (milliseconds)=3196
12/05/09 02:35:06 INFO mapred.JobClient:     Shutdown (milliseconds)=124
12/05/09 02:35:06 INFO mapred.JobClient:     Vertex input superstep (milliseconds)=13130
12/05/09 02:35:06 INFO mapred.JobClient:     Superstep 0 (milliseconds)=8564
12/05/09 02:35:06 INFO mapred.JobClient:     Superstep 4 (milliseconds)=5540
12/05/09 02:35:06 INFO mapred.JobClient:     Superstep 5 (milliseconds)=2012
12/05/09 02:35:06 INFO mapred.JobClient:     Superstep 2 (milliseconds)=8601
12/05/09 02:35:06 INFO mapred.JobClient:     Superstep 1 (milliseconds)=6271

These results are fairly similar to the first set (even though there are more workers).  I'm pretty sure we can squeeze more performance from Netty in the future in future patches (i.e. local send optimization is missing, tuning TCP parameters, exposing more knobs to the user, etc.).
                
> Implement Netty-backed rpc solution
> -----------------------------------
>
>                 Key: GIRAPH-37
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-37
>             Project: Giraph
>          Issue Type: New Feature
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: GIRAPH-37-wip.patch, GIRAPH-37.patch
>
>
> GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira