Return-Path: X-Original-To: apmail-incubator-giraph-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-giraph-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7D86FC821 for ; Wed, 9 May 2012 10:10:49 +0000 (UTC) Received: (qmail 11730 invoked by uid 500); 9 May 2012 10:10:49 -0000 Delivered-To: apmail-incubator-giraph-dev-archive@incubator.apache.org Received: (qmail 11650 invoked by uid 500); 9 May 2012 10:10:48 -0000 Mailing-List: contact giraph-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: giraph-dev@incubator.apache.org Delivered-To: mailing list giraph-dev@incubator.apache.org Received: (qmail 11622 invoked by uid 99); 9 May 2012 10:10:48 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 May 2012 10:10:48 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 9E7351C3F40; Wed, 9 May 2012 10:10:46 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============8006324818854538054==" MIME-Version: 1.0 Subject: Re: Review Request: Implemented a netty client/server protocol a a faster alternative to Hadoop RPC (3x improvement) From: "Sebastian Schelter" To: "Sebastian Schelter" , "Avery Ching" , "giraph" Date: Wed, 09 May 2012 10:10:46 -0000 Message-ID: <20120509101046.1539.86635@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org X-ReviewRequest-URL: https://reviews.apache.org/r/5074/ In-Reply-To: <20120509092236.1511.25932@reviews.apache.org> References: <20120509092236.1511.25932@reviews.apache.org> --===============8006324818854538054== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5074/#review7728 ----------------------------------------------------------- Ship it! I went through the code (although I don't have much experience with network= ing code), everything looks very well. I tested this patch by computing the connected components of the undirected= wikipedia pagelink graph (6M vertices, 250M edges) on a 6 machine cluster.= Everything went fine and I even saw a small improvement in runtime althoug= h the job only takes 4 minutes. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/ap= ache/giraph/graph/BspServiceWorker.java I don't like it that a collection is changed outside of the class that = owns it. = = This makes code hard to read and debug. We should rather introduce a me= thod for this in the class that owns this map to have all mutations in one = place. - Sebastian On 2012-05-09 09:22:36, Avery Ching wrote: > = > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/5074/ > ----------------------------------------------------------- > = > (Updated 2012-05-09 09:22:36) > = > = > Review request for giraph. > = > = > Summary > ------- > = > * Implemented a request/response protocol with netty as a NettyClient and= NettyServer. There is a NettyClientWorker and NettyClientServer that impl= ements WorkerClient and WorkerServer, respectively. Netty is a lot faster = since it's non-blocking and we can interleave computation and communication= as opposed to Hadoop RPC (blocking). > * The netty server implementation uses concurrent hash maps to improved c= oncurrency instead of synchronized blocks around maps. > * By default netty is used, but Hadoop RPC can be used with -Dgiraph.useN= etty=3Dfalse > * Changed the class hierarchy of ServerInterface to WorkerClientServer (W= orkerClient and WorkerServer) to support a request/response protocol instea= d of just RPC > * In netty, the messages/mutations are gathered by partition and send out= as a partition's worth of messages/mutations > * Added two new test classes (RequestTest.java and ConnectionTest.java) t= o test all requests and check netty connections. > * PageRankBenchmark uses EdgeListVertex as a default > = > = > This addresses bug GIRAPH-37. > https://issues.apache.org/jira/browse/GIRAPH-37 > = > = > Diffs > ----- > = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/or= g/apache/giraph/utils/MockUtils.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/or= g/apache/giraph/examples/SimpleShortestPathVertexTest.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/or= g/apache/giraph/comm/RequestTest.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/graph/WorkerInfo.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/or= g/apache/giraph/TestJsonBase64Format.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/or= g/apache/giraph/TestManualCheckpoint.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/or= g/apache/giraph/comm/ConnectionTest.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/graph/GiraphJob.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/graph/GraphState.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/graph/VertexMutations.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/WritableRequest.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/graph/BspService.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/graph/BspServiceWorker.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/WorkerClientServer.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/WorkerCommunications.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/WorkerServer.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/WorkerClient.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/ServerInterface.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/ServerData.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/SendVertexRequest.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/SendPartitionMutationsRequest.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/SendPartitionMessagesRequest.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/SendMutationsCache.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/SendMessageCache.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/RequestServerHandler.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/ResponseClientHandler.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/RequestRegistry.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/RequestEncoder.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/RequestDecoder.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/NettyWorkerServer.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/NettyWorkerClientServer.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/NettyWorkerClient.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/NettyServer.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/NettyClient.java PRE-CREATION = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/comm/BasicRPCCommunications.java 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1332888 = > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/or= g/apache/giraph/benchmark/PageRankBenchmark.java 1332888 = > = > Diff: https://reviews.apache.org/r/5074/diff > = > = > Testing > ------- > = > 'mvn verify' passes. I ran several test runs to gather performance resul= ts. Here is a simple example: > = > Hadoop RPC: > hadoop jar ~/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar org.apache.gir= aph.benchmark.PageRankBenchmark -Dgiraph.useNetty=3Dfalse -w 5 -V 5000000 -= s 5 -e 2 -v > = > 12/05/09 01:59:56 INFO mapred.JobClient: Giraph Timers > 12/05/09 01:59:56 INFO mapred.JobClient: Total (milliseconds)=3D167722 > 12/05/09 01:59:56 INFO mapred.JobClient: Superstep 3 (milliseconds)= =3D24775 > 12/05/09 01:59:56 INFO mapred.JobClient: Setup (milliseconds)=3D2930 > 12/05/09 01:59:56 INFO mapred.JobClient: Shutdown (milliseconds)=3D181 > 12/05/09 01:59:56 INFO mapred.JobClient: Vertex input superstep (mill= iseconds)=3D51025 > 12/05/09 01:59:56 INFO mapred.JobClient: Superstep 0 (milliseconds)= =3D21543 > 12/05/09 01:59:56 INFO mapred.JobClient: Superstep 4 (milliseconds)= =3D19858 > 12/05/09 01:59:56 INFO mapred.JobClient: Superstep 5 (milliseconds)= =3D2844 > 12/05/09 01:59:56 INFO mapred.JobClient: Superstep 2 (milliseconds)= =3D24507 > 12/05/09 01:59:56 INFO mapred.JobClient: Superstep 1 (milliseconds)= =3D20054 > = > Netty: > hadoop jar ~/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar org.apache.gir= aph.benchmark.PageRankBenchmark -Dgiraph.useNetty=3Dtrue -w 5 -V 5000000 -s= 5 -e 2 -v > = > 12/05/09 02:06:10 INFO mapred.JobClient: Giraph Timers > 12/05/09 02:06:10 INFO mapred.JobClient: Total (milliseconds)=3D57795 > 12/05/09 02:06:10 INFO mapred.JobClient: Superstep 3 (milliseconds)= =3D7636 > 12/05/09 02:06:10 INFO mapred.JobClient: Setup (milliseconds)=3D3574 > 12/05/09 02:06:10 INFO mapred.JobClient: Shutdown (milliseconds)=3D232 > 12/05/09 02:06:10 INFO mapred.JobClient: Vertex input superstep (mill= iseconds)=3D13393 > 12/05/09 02:06:10 INFO mapred.JobClient: Superstep 0 (milliseconds)= =3D5610 > 12/05/09 02:06:10 INFO mapred.JobClient: Superstep 4 (milliseconds)= =3D8473 > 12/05/09 02:06:10 INFO mapred.JobClient: Superstep 5 (milliseconds)= =3D1844 > 12/05/09 02:06:10 INFO mapred.JobClient: Superstep 2 (milliseconds)= =3D7418 > 12/05/09 02:06:10 INFO mapred.JobClient: Superstep 1 (milliseconds)= =3D9612 > = > These were some median runs. The overall runtime improved from 167722 -> = 57795 with Netty (2.9x faster). Loading the vertices improved from 51025 -= > 13393 (3.8x faster). More results coming tomorrow, but for bigger runs, = the improvement is likely to be even more than 3x. > = > = > Thanks, > = > Avery > = > --===============8006324818854538054==--