Mailing-List: contact dev-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hama.apache.org
Date: Fri, 15 Feb 2013 13:05:14 +0000 (UTC)
From: "Suraj Menon (JIRA)" <jira@apache.org>
To: dev@hama.apache.org
Message-ID: <JIRA.12625550.1356926049028.293151.1360933514349@arcas>
In-Reply-To: <JIRA.12625550.1356926049028@arcas>
References: <JIRA.12625550.1356926049028@arcas>
Subject: [jira] [Commented] (HAMA-704) Optimization of memory usage during
 message processing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HAMA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579152#comment-13579152 ] 

Suraj Menon commented on HAMA-704:
----------------------------------

Yes! Sorting would take more time, especially till messaging is synchronous. 
I am not an expert, but can we call compute before aggregation by master?

As we decided, you can check this on the new branch.

On a side note, have you profiled and checked the number of objects created in the process? Every compare in SortedQueue involves creating new String object. Can you check if removing  and doing VertexId.compare(otherVertexId) makes any difference. 

I would suggest that the requirements get defined here. The design should get things to work on 16GB 400 node cluster as well as a 4GB 15 node cluster. In any scenario, if we reach the main memory limit, we would have to spill to disk or find some other modes of persistence. 
                
> Optimization of memory usage during message processing
> ------------------------------------------------------
>
>                 Key: HAMA-704
>                 URL: https://issues.apache.org/jira/browse/HAMA-704
>             Project: Hama
>          Issue Type: Improvement
>          Components: graph
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>            Priority: Critical
>             Fix For: 0.6.1
>
>         Attachments: hama-704_v05.patch, localdisk.patch, mytest.patch, patch.txt, patch.txt, removeMsgMap.patch
>
>
> <vertex, message> map seems consume a lot of memory. We should figure out an efficient way to reduce memory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira