Return-Path: X-Original-To: apmail-incubator-giraph-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-giraph-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F370F9E68 for ; Wed, 16 Nov 2011 22:37:46 +0000 (UTC) Received: (qmail 28798 invoked by uid 500); 16 Nov 2011 22:37:46 -0000 Delivered-To: apmail-incubator-giraph-dev-archive@incubator.apache.org Received: (qmail 28754 invoked by uid 500); 16 Nov 2011 22:37:46 -0000 Mailing-List: contact giraph-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: giraph-dev@incubator.apache.org Delivered-To: mailing list giraph-dev@incubator.apache.org Received: (qmail 28744 invoked by uid 99); 16 Nov 2011 22:37:46 -0000 Received: from reviews.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Nov 2011 22:37:46 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id BA64F1C2C4B; Wed, 16 Nov 2011 22:37:46 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============6700178226589795204==" MIME-Version: 1.0 Subject: Review Request: GIRAPH-91 - Large-memory improvements (Memory reduced vertex implementation, fast failure, added settings) From: "Avery Ching" To: "Avery Ching" , "giraph" Date: Wed, 16 Nov 2011 22:37:46 -0000 Message-ID: <20111116223746.20719.90906@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org X-ReviewRequest-URL: https://reviews.apache.org/r/2868/ --===============6700178226589795204== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2868/ ----------------------------------------------------------- Review request for giraph. Summary ------- There general changes should support larger heap sizes (i.e. >20G) - Added new EdgeListVertex that stores its edges in a compact pair of lists= instead of Vertex's HashMap. - Added unittests TestEdgeArrayVertex to test EdgeListVertex. - Augmented PageRankBenchmark to choose between EdgeListArrayVertex or Vert= ex (to try it out). - Added failure cleanup for failed workers to quickly alert the master that= they are dead by deleting its health ephemeral znode. This allows us to s= et higher ZooKeeper timeouts to deal with GC pauses and the like. In a qui= ck test of 3 nodes, I saw failure in 43 seconds instead of 1m 52 sec. - Added a context.progress() to flushing to not kill jobs with long timeout= s (GC or lots of messages). This addresses bug GIRAPH-91. https://issues.apache.org/jira/browse/GIRAPH-91 Diffs ----- http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/benchmark/PageRankBenchmark.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/bsp/CentralizedServiceWorker.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/comm/BasicRPCCommunications.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/graph/BspServiceWorker.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/graph/EdgeListVertex.java PRE-CREATION = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/graph/GiraphJob.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/= apache/giraph/graph/GraphMapper.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/= apache/giraph/TestJsonBase64Format.java 1202898 = http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/= apache/giraph/graph/TestEdgeListVertex.java PRE-CREATION = Diff: https://reviews.apache.org/r/2868/diff Testing ------- Local unittests, PageRankBenchmark on multiple machines with >20GB heaps. Thanks, Avery --===============6700178226589795204==--