incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-104) Save half of maximum memory used from messaging
Date Wed, 14 Dec 2011 01:02:30 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168938#comment-13168938
] 

Avery Ching commented on GIRAPH-104:
------------------------------------

By the way, here's example output from the changes to RandomMessageBenchmark.  It will help
us qualify messaging improvements.

2011-12-12 23:58:54,887 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Outputing statistics for superstep 4
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
superstep total bytes sent : 60000000000
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
total bytes sent : 240000000000
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
superstep total messages : 6000000
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
total messages : 24000000
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
superstep total millis : 854309
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
total millis : 3718123
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
workers : 5
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Superstep megabytes / second = 334.8932235547969
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Total megabytes / second = 307.7921789267058
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Superstep messages / second = 35116.09967821947
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Total messages / second = 32274.349181024943
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Superstep megaabytes / second / worker = 66.97864471095939
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Total megabytes / second / worker = 61.55843578534116
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Superstep messages / second / worker = 7023.219935643894
2011-12-12 23:58:54,888 INFO org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
Total messages / second / worker = 6454.869836204989
2011-12-12 23:58:57,627 INFO org.apache.giraph.comm.BasicRPCCommunications: flush: starting
for superstep 4 totalMem = 20463.375M, maxMem = 20463.375M, freeMem = 6571.4233M

                
> Save half of maximum memory used from messaging
> -----------------------------------------------
>
>                 Key: GIRAPH-104
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-104
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>            Priority: Critical
>         Attachments: GIRAPH-104.diff
>
>
> Currently, the amount of memory that Giraph uses for messaging is huge.  This JIRA will
reduce the messaging memory by half and provide periodic updates of memory for debugging.
 Details are below:
> Refactored RandomMessageBenchmark to an internal vertex class.  Added aggregators to
RandomMessagesBenchmark to track bytes, messages, and time for the messaging.  Adjusted the
postSuperstep() to be called after the flush() for more accurate timings.
> Added periodic minute updates for message flushing (which can take a while, especially
on the memory benchmark).  This helps to see how progress is going and gives an ETA.
> Memory optimizations include:
> - Clear the message list after computation 
> - Free vertex messages on the source as the flush is going on 
> - TreeMap -> HashMap for VertexMutations
> - Sizing the ArrayList properly in transientInMessages

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message