hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-704) Optimization of memory usage during message processing
Date Tue, 19 Feb 2013 20:35:13 GMT

    [ https://issues.apache.org/jira/browse/HAMA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581601#comment-13581601
] 

Thomas Jungblut commented on HAMA-704:
--------------------------------------

bq.So, I expect that we can reduce memory consumption by removing Map<Vertex, List<M>>
and using of spilling queue.

Another consumer of memory is the MapWritable in the GraphJobMessage. It would be nice if
this can be replaced with a stream iterator once there is the means for sorted/merged queues.
Then the messages can be iterated in ascending order of the vertexid and easily be joined
with the vertices in whatever storage we have without the need for random access.
                
> Optimization of memory usage during message processing
> ------------------------------------------------------
>
>                 Key: HAMA-704
>                 URL: https://issues.apache.org/jira/browse/HAMA-704
>             Project: Hama
>          Issue Type: Improvement
>          Components: graph
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>            Priority: Critical
>             Fix For: 0.6.1
>
>         Attachments: HAMA-704.patch-v1, hama-704_v05.patch, localdisk.patch, mytest.patch,
patch.txt, patch.txt, removeMsgMap.patch
>
>
> <vertex, message> map seems consume a lot of memory. We should figure out an efficient
way to reduce memory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message