giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjin Lee <dongjin.lee...@gmail.com>
Subject Re: What is the best recipe for determining maximum worker count?
Date Wed, 11 Feb 2015 06:12:33 GMT
OK. I will take more experiments and let you know.

Thanks,
Dongjin

On Mon, Feb 9, 2015 at 4:46 AM, Eli Reisman <apache.mailbox@gmail.com>
wrote:

> That's correct. There were some changes last year as to. How these worker
> containers are allocated that were supposed to fix another similar issue
> with the master task. Perhaps this introduced another bug. I'm not sure
> there's enough info to be sure can you give more details? Anyone else
> seeing this issue? And to confirm this was only when you run on a single
> machine?
> On Jan 17, 2015 12:09 PM, "Claudio Martella" <claudio.martella@gmail.com>
> wrote:
>
> > as far as I understand, with YARN there are two additional task to the
> > workers: one for the master and one for the application manager. I'm not
> > sure, but maybe Eli can confirm.
> >
> > On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <dongjin.lee.kr@gmail.com>
> > wrote:
> >
> > > * I apologize in advance for my bad English.
> > >
> > > Hello. I have a question about determining optimal worker count. I have
> > > been trying SimpleShortestPathsComputation example with Hadoop 2.5.1.
> And
> > > noticed that the maximum value for -w parameter differs per
> > configuration.
> > >
> > > Let me explain: I have been running Giraph jobs on my local Ubuntu
> > machine,
> > > with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads
> > simultaneously
> > > and has 9gb of free memory.
> > >
> > > First, I configured Hadoop can run 4 mappers with 2gb of memory for
> each.
> > > When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did
> > not
> > > work - It froze with 'map 75%'. That is, I could not use 2 mappers for
> > > worker.
> > >
> > > After that, I re-configured my Hadoop distribution to run 8 mappers
> with
> > > 1gb of memory for each. When I re-ran SimpleShortestPathsComputation,
> w=5
> > > worked well but with w=6 it froze with 'map 86%'. In this case, I could
> > not
> > > use 3 mappers for worker.
> > >
> > > I already know that there is an additional mapper, which works as
> > > BSPMaster. However, It is still mysterious for me: Is there any hidden
> > > process I don't know? Then, how many are there? 1 or 2?
> > >
> > > Any guidance will be welcomed.
> > >
> > > Thanks in advance,
> > > Dongjin
> > >
> > > PS - When I was working with Hadoop 1.x.x, I calculated the maximum
> > worker
> > > count as {available mappers} - 1 and it worked perfectly. I guess there
> > are
> > > some differences between 1.x.x and 2.x.x that I don't know.
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > >
> > > *Oracle Server Technologies Group.So interested in massive-scale
> machine
> > > learning.facebook: www.facebook.com/dongjin.lee.kr
> > > <http://www.facebook.com/dongjin.lee.kr>linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>github:
> > > <http://goog_969573159>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> > > <http://www.twitter.com/dongjinleekr>*
> > >
> >
> >
> >
> > --
> >    Claudio Martella
> >
>



-- 
*Dongjin Lee*


*Oracle Server Technologies Group.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message