hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HAMA-704) Optimization of memory usage during message processing
Date Fri, 15 Feb 2013 10:35:12 GMT

    [ https://issues.apache.org/jira/browse/HAMA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579077#comment-13579077
] 

Edward J. Yoon edited comment on HAMA-704 at 2/15/13 10:33 AM:
---------------------------------------------------------------

My question is whether this is really helpful. According to my simple Java test, it looks
like helpful. 1) program throws OOM exception but 2) is ok.

{code}
// 1. current TRUNK

Map<Integer, List<Integer>> map;
for(int i = 0; i < 1000000000; i++) {
  msg = new ArrayList<Integer>();
  for(int j = 0; j < 100000; j++) {
    msg.add(j);
  }

  map.put(i, msg);
}
{code}

vs. 

{code}
// 1. call compute() within loop

for(int i = 0; i < 1000000000; i++) {
  msg = new ArrayList<Integer>();
  for(int j = 0; j < 100000; j++) {
    msg.add(j);
  }

  // i.compute(msg);
}

{code}

And, I couldn't test with huge graph yet but, performance will be decreased by cost of sort.

With my single machine, Pagerank on 1,600,000 edges graph:

TRUNK 12 secs
TRUNK + SortedMessageQueue 39 secs
--
My patch and + SortedMessageQueue 39 secs.

                
      was (Author: udanax):
    My question is that this is really helpful. I couldn't test with huge graph yet.

According to my simple Java test, it looks like helpful. 1) program throws OOM exception but
2) is ok.

{code}
// 1. current TRUNK

Map<Integer, List<Integer>> map;
for(int i = 0; i < 1000000000; i++) {
  msg = new ArrayList<Integer>();
  for(int j = 0; j < 100000; j++) {
    msg.add(j);
  }

  map.put(i, msg);
}
{code}

vs. 

{code}
// 1. call compute() within loop

for(int i = 0; i < 1000000000; i++) {
  msg = new ArrayList<Integer>();
  for(int j = 0; j < 100000; j++) {
    msg.add(j);
  }

  // i.compute(msg);
}

{code}
                  
> Optimization of memory usage during message processing
> ------------------------------------------------------
>
>                 Key: HAMA-704
>                 URL: https://issues.apache.org/jira/browse/HAMA-704
>             Project: Hama
>          Issue Type: Improvement
>          Components: graph
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>            Priority: Critical
>             Fix For: 0.6.1
>
>         Attachments: hama-704_v05.patch, localdisk.patch, mytest.patch, patch.txt, patch.txt,
removeMsgMap.patch
>
>
> <vertex, message> map seems consume a lot of memory. We should figure out an efficient
way to reduce memory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message