giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: What is the best recipe for determining maximum worker count?
Date Sun, 08 Feb 2015 19:46:35 GMT
That's correct. There were some changes last year as to. How these worker
containers are allocated that were supposed to fix another similar issue
with the master task. Perhaps this introduced another bug. I'm not sure
there's enough info to be sure can you give more details? Anyone else
seeing this issue? And to confirm this was only when you run on a single
machine?
On Jan 17, 2015 12:09 PM, "Claudio Martella" <claudio.martella@gmail.com>
wrote:

> as far as I understand, with YARN there are two additional task to the
> workers: one for the master and one for the application manager. I'm not
> sure, but maybe Eli can confirm.
>
> On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <dongjin.lee.kr@gmail.com>
> wrote:
>
> > * I apologize in advance for my bad English.
> >
> > Hello. I have a question about determining optimal worker count. I have
> > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And
> > noticed that the maximum value for -w parameter differs per
> configuration.
> >
> > Let me explain: I have been running Giraph jobs on my local Ubuntu
> machine,
> > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads
> simultaneously
> > and has 9gb of free memory.
> >
> > First, I configured Hadoop can run 4 mappers with 2gb of memory for each.
> > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did
> not
> > work - It froze with 'map 75%'. That is, I could not use 2 mappers for
> > worker.
> >
> > After that, I re-configured my Hadoop distribution to run 8 mappers with
> > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5
> > worked well but with w=6 it froze with 'map 86%'. In this case, I could
> not
> > use 3 mappers for worker.
> >
> > I already know that there is an additional mapper, which works as
> > BSPMaster. However, It is still mysterious for me: Is there any hidden
> > process I don't know? Then, how many are there? 1 or 2?
> >
> > Any guidance will be welcomed.
> >
> > Thanks in advance,
> > Dongjin
> >
> > PS - When I was working with Hadoop 1.x.x, I calculated the maximum
> worker
> > count as {available mappers} - 1 and it worked perfectly. I guess there
> are
> > some differences between 1.x.x and 2.x.x that I don't know.
> >
> > --
> > *Dongjin Lee*
> >
> >
> > *Oracle Server Technologies Group.So interested in massive-scale machine
> > learning.facebook: www.facebook.com/dongjin.lee.kr
> > <http://www.facebook.com/dongjin.lee.kr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > <http://kr.linkedin.com/in/dongjinleekr>github:
> > <http://goog_969573159>github.com/dongjinleekr
> > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> > <http://www.twitter.com/dongjinleekr>*
> >
>
>
>
> --
>    Claudio Martella
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message