hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuwenhua <xuwen...@huawei.com>
Subject 答复: 答复: About hama performance
Date Thu, 19 Dec 2013 09:06:26 GMT
Hi Edward,
I only tested pagerank.

Another problem is “OutOfMem”. The test environment is the same as I mentioned in last
mail.
For hama, I tested with 0.4M vertex with 100 edges using 20 task, it shows “OutOfMem”.
And the local file size is about 260M.
But for giraph, I tested with 10M vertex with 10 edges using 19 task, it can run well. And
the local file size is about 830M.

So do you encounter the same problem and how to avoid it?? I think the test data is not very
large.


----------------------------------------------------------------------------------------------------------------------------------------------------------
Wenhua XU
Research Dept. of Software Co., Ltd
[cid:image001.jpg@01C802B8.FA7E5340]  To enrich life through communication
Office: +86-755-28970151    Fax: +86-755-28975077
----------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!
----------------------------------------------------------------------------------------------------------------------------------------------------------

发件人: Edward J. Yoon [mailto:edwardyoon@apache.org]
发送时间: 2013年12月13日 19:29
收件人: Xuwenhua
抄送: user@hama.apache.org; dev@hama.apache.org
主题: Re: 答复: About hama performance

> According to your message, I think this issue shall be high priority, right?? So which
version and when will it be fixed and released??

Yes, it's on my todo list. I plan to fix before releasing version 0.7. BTW, have you compared
the performance of SSSP or K-Means? You'll see another result.

I'm CC'ing user@h.a.o<mailto:user@h.a.o> and dev@h.a.o<mailto:dev@h.a.o> lists.
On Fri, Dec 13, 2013 at 7:26 PM, Xuwenhua <xuwenhua@huawei.com<mailto:xuwenhua@huawei.com>>
wrote:
Hi Edward,
Thank you for your information.

You mean I re-send this email in hama user@ list? Or you copy it to list directly??
For I do not subscribe the list now, so if need I re-send, I will send it next week. If you
can directly copy it, and you can do it.

According to your message, I think this issue shall be high priority, right?? So which version
and when will it be fixed and released??

----------------------------------------------------------------------------------------------------------------------------------------------------------
Wenhua XU
Research Dept. of Software Co., Ltd
[cid:image001.jpg@01C802B8.FA7E5340]  To enrich life through communication
Office: +86-755-28970151<tel:%2B86-755-28970151>    Fax: +86-755-28975077<tel:%2B86-755-28975077>
----------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!
----------------------------------------------------------------------------------------------------------------------------------------------------------

发件人: Edward J. Yoon [mailto:edwardyoon@apache.org<mailto:edwardyoon@apache.org>]
发送时间: 2013年12月13日 17:47
收件人: Xuwenhua
主题: Re: About hama performance

Hi Xuwenhua,

> From the test report, we can see that hama cost twice time.

I guess you did correctly. Hama's Graph package is still under heavy development.

Currently Hama uses additional supersteps to broadcast aggregators, and sends messages twice
(https://issues.apache.org/jira/browse/HAMA-734). It means that the performance will be improved
by at least 50%.

BTW, can we move this to Hama user@ list?

On Fri, Dec 13, 2013 at 5:08 PM, Xuwenhua <xuwenhua@huawei.com<mailto:xuwenhua@huawei.com>>
wrote:
Dear Edward,
This is Wenhua From Huawei. We are interested in Hama for it BSP feature.
And those days I study some hama material and do some performance tests compared with Giraph.

I used 3 servers with total core 20 as follows, and 1G for “bsp.child.java.opts”
CPU(8 core) * 1, 8G*6   master/name node,slave/data node
CPU(8 core)* 1,4G*6            slave/data node
CPU(2 core) * 2,2G*4            slave/data node
And the input format is like “vertex neighbour1 neighbour2”.

I tested from 0.1m vertex with 10 edges to 1m vertex with 10 edges.
Giraph using 19 task and Hama using 20 task, so that each core can deal with one task.
I test the pagerank, I force it to run 30 iterations,
      if(this.getSuperstepCount() >= 30)
      {
         voteToHalt();
          return;
      }
And the test result is as follows.
[cid:image003.png@01CEF818.4E9CDE50]
From the test report, we can see that hama cost twice time.

So my query is that is there any problem in my experiments??
And do you have some hama performance information compared with Giraph?? Because I only see
some hama’s own performance in hama website.

Looking forward to receiving your comments.

Best Regards


----------------------------------------------------------------------------------------------------------------------------------------------------------
Wenhua XU
Research Dept. of Software Co., Ltd
[cid:image001.jpg@01C802B8.FA7E5340]  To enrich life through communication
Office: +86-755-28970151<tel:%2B86-755-28970151>    Fax: +86-755-28975077<tel:%2B86-755-28975077>
----------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!
----------------------------------------------------------------------------------------------------------------------------------------------------------




--
Best Regards, Edward J. Yoon
@eddieyoon



--
Best Regards, Edward J. Yoon
@eddieyoon
Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message