giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claudio Martella (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-461) Convert static assignment of in-memory partitions with LRU cache
Date Tue, 05 Feb 2013 11:45:29 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13571246#comment-13571246
] 

Claudio Martella commented on GIRAPH-461:
-----------------------------------------

Ran this multiple times. Looks like performance of trunk when OOC is a bit slower. I would
not consider this very reliable, let's say the behave the same, as expected on PR. I'm committing
this one.


13/02/05 12:37:49 INFO mapred.JobClient:   Giraph Timers
13/02/05 12:37:49 INFO mapred.JobClient:     Total (milliseconds)=585851
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 3 (milliseconds)=56282
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 4 (milliseconds)=38479
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 10 (milliseconds)=6282
13/02/05 12:37:49 INFO mapred.JobClient:     Setup (milliseconds)=20225
13/02/05 12:37:49 INFO mapred.JobClient:     Shutdown (milliseconds)=139
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 7 (milliseconds)=48529
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 9 (milliseconds)=47064
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 0 (milliseconds)=55454
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 8 (milliseconds)=52432
13/02/05 12:37:49 INFO mapred.JobClient:     Input superstep (milliseconds)=64258
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 6 (milliseconds)=52334
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 5 (milliseconds)=48943
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 2 (milliseconds)=42660
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep 1 (milliseconds)=52766
13/02/05 12:37:49 INFO mapred.JobClient:   Giraph Stats
13/02/05 12:37:49 INFO mapred.JobClient:     Aggregate edges=1000000000
13/02/05 12:37:49 INFO mapred.JobClient:     Superstep=11
13/02/05 12:37:49 INFO mapred.JobClient:     Last checkpointed superstep=0
13/02/05 12:37:49 INFO mapred.JobClient:     Current workers=60
13/02/05 12:37:49 INFO mapred.JobClient:     Current master task partition=0
13/02/05 12:37:49 INFO mapred.JobClient:     Sent messages=0
13/02/05 12:37:49 INFO mapred.JobClient:     Aggregate finished vertices=10000000
13/02/05 12:37:49 INFO mapred.JobClient:     Aggregate vertices=10000000
13/02/05 12:37:49 INFO mapred.JobClient:   File Output Format Counters 
13/02/05 12:37:49 INFO mapred.JobClient:     Bytes Written=0
13/02/05 12:37:49 INFO mapred.JobClient:   FileSystemCounters
13/02/05 12:37:49 INFO mapred.JobClient:     HDFS_BYTES_READ=2684
13/02/05 12:37:49 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=1397805
13/02/05 12:37:49 INFO mapred.JobClient:   File Input Format Counters 
13/02/05 12:37:49 INFO mapred.JobClient:     Bytes Read=0
13/02/05 12:37:49 INFO mapred.JobClient:   Map-Reduce Framework
13/02/05 12:37:49 INFO mapred.JobClient:     Map input records=61
13/02/05 12:37:49 INFO mapred.JobClient:     Physical memory (bytes) snapshot=68512210944
13/02/05 12:37:49 INFO mapred.JobClient:     Spilled Records=0
13/02/05 12:37:49 INFO mapred.JobClient:     CPU time spent (ms)=17221490
13/02/05 12:37:49 INFO mapred.JobClient:     Total committed heap usage (bytes)=58657275904
13/02/05 12:37:49 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=371997724672
13/02/05 12:37:49 INFO mapred.JobClient:     Map output records=0
13/02/05 12:37:49 INFO mapred.JobClient:     SPLIT_RAW_BYTES=2684

                
> Convert static assignment of in-memory partitions with LRU cache
> ----------------------------------------------------------------
>
>                 Key: GIRAPH-461
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-461
>             Project: Giraph
>          Issue Type: Sub-task
>          Components: graph
>            Reporter: Claudio Martella
>            Assignee: Claudio Martella
>         Attachments: GIRAPH-461.patch, GIRAPH-461.patch, GIRAPH-461.patch
>
>
> Currently, the out-of-core partitions are assigned to memory or to disk statically. Using
an LRU cache should help keeping in-memory only the partitions that are actively accessed,
given a job that does not access all the graph at each superstep (traversals) and a good data
partitioning (non random).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message