giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjin Lee <dongjin.lee...@gmail.com>
Subject What is the best recipe for determining maximum worker count?
Date Fri, 16 Jan 2015 13:16:50 GMT
* I apologize in advance for my bad English.

Hello. I have a question about determining optimal worker count. I have
been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And
noticed that the maximum value for -w parameter differs per configuration.

Let me explain: I have been running Giraph jobs on my local Ubuntu machine,
with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads simultaneously
and has 9gb of free memory.

First, I configured Hadoop can run 4 mappers with 2gb of memory for each.
When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did not
work - It froze with 'map 75%'. That is, I could not use 2 mappers for
worker.

After that, I re-configured my Hadoop distribution to run 8 mappers with
1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5
worked well but with w=6 it froze with 'map 86%'. In this case, I could not
use 3 mappers for worker.

I already know that there is an additional mapper, which works as
BSPMaster. However, It is still mysterious for me: Is there any hidden
process I don't know? Then, how many are there? 1 or 2?

Any guidance will be welcomed.

Thanks in advance,
Dongjin

PS - When I was working with Hadoop 1.x.x, I calculated the maximum worker
count as {available mappers} - 1 and it worked perfectly. I guess there are
some differences between 1.x.x and 2.x.x that I don't know.

-- 
*Dongjin Lee*


*Oracle Server Technologies Group.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message