Return-Path: X-Original-To: apmail-giraph-dev-archive@www.apache.org Delivered-To: apmail-giraph-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8580DD203 for ; Thu, 16 Aug 2012 16:08:10 +0000 (UTC) Received: (qmail 97137 invoked by uid 500); 16 Aug 2012 16:08:10 -0000 Delivered-To: apmail-giraph-dev-archive@giraph.apache.org Received: (qmail 97091 invoked by uid 500); 16 Aug 2012 16:08:09 -0000 Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@giraph.apache.org Delivered-To: mailing list dev@giraph.apache.org Delivered-To: moderator for dev@giraph.apache.org Received: (qmail 1449 invoked by uid 99); 16 Aug 2012 12:42:58 -0000 Content-Type: multipart/alternative; boundary="===============3680509687032416055==" MIME-Version: 1.0 Subject: Re: Review Request: Move part of the graph out-of-core when memory is low From: "Maja Kabiljo" To: "Avery Ching" Cc: "Maja Kabiljo" , "giraph" , "Alessandro Presta" Date: Thu, 16 Aug 2012 12:42:56 -0000 Message-ID: <20120816124256.23841.81084@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Maja Kabiljo" X-ReviewGroup: giraph X-ReviewRequest-URL: https://reviews.apache.org/r/5987/ X-Sender: "Maja Kabiljo" References: <20120815165157.23839.98602@reviews.apache.org> In-Reply-To: <20120815165157.23839.98602@reviews.apache.org> Reply-To: "Maja Kabiljo" --===============3680509687032416055== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/#review10400 ----------------------------------------------------------- I'll take a deeper look into this once you have a final version, for now on= e comment: When we do checkpointing, there is no need to deserialize all th= e partitions and then write them again to checkpoint file, instead you coul= d just copy the bytes from out-of-core files (I did something similar with = message stores if you want to take a look). Another thing, would it be possible to make some kind of optimization for n= on-mutating graphs - to keep the edges on disk all the time and just rewrit= e vertex values? - Maja Kabiljo On Aug. 15, 2012, 4:51 p.m., Alessandro Presta wrote: > = > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/5987/ > ----------------------------------------------------------- > = > (Updated Aug. 15, 2012, 4:51 p.m.) > = > = > Review request for giraph and Avery Ching. > = > = > Description > ------- > = > I gave this another shot. This time it plays nicely with input superstep:= I replaced both the temporary partitions and the worker partition map with= a common data structure, PartitionStore, held in ServerData. This is simil= ar to what we now have for messages. > = > Partition is now thread-safe so that we can have concurrent calls to putV= ertex(). > = > SimplePartitionStore is backed by a concurrent hash map (nothing new here= , except that we skip copying partitions to the worker). > = > DiskBackedPartitionStore can hold up to a user-defined number of partitio= ns in memory, and spill the remaining ones to disk. Each partition is store= d in a separate file. > Adding vertices to an out-of-core partition consists in appending them to= the file, which makes processing vertex requests relatively fast. > We use a ReadWriteLock for each partition: performing operations on a par= tition held in memory only requires a read-lock (since Partition is thread-= safe), while creating a new partition, moving it in and out of core or appe= nding vertices requires a write-lock (we can't have concurrent writes). > = > Also note that this breaks Hadoop RPC: I preferred to keep it clean (this= also shows what code we get rid of) instead of working around it. I suppos= e the Netty security patch will be completed soon. If not, I will restore R= PC compatibility. > = > More here: https://issues.apache.org/jira/browse/GIRAPH-249?focusedCommen= tId=3D13435280&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comme= nt-tabpanel#comment-13435280 > = > = > This addresses bug GIRAPH-249. > https://issues.apache.org/jira/browse/GIRAPH-249 > = > = > Diffs > ----- > = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/bsp/CentralizedServiceWorker.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/BasicRPCCommunications.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/NettyWorkerClientServer.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/NettyWorkerServer.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/SendVertexRequest.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/ServerData.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/WorkerServer.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/BspServiceWorker.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/GiraphJob.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/GraphMapper.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/DiskBackedPartitionStore.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/HashWorkerPartitioner.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/Partition.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/PartitionStore.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/SimplePartitionStore.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/WorkerGraphPartitioner.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestMutateGraphVertex.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/ConnectionTest.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/RequestTest.java 1373331 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/graph/partition/TestPartitionStores.java PRE-CREATION = > = > Diff: https://reviews.apache.org/r/5987/diff/ > = > = > Testing > ------- > = > mvn verify and pseudo-distributed mode tests with both SimplePartitionSto= re and DiskBackedPartitionStore > = > = > Thanks, > = > Alessandro Presta > = > --===============3680509687032416055==--