giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco Sclano <sclano.france...@gmail.com>
Subject Re: how determine the number of workers to set in -w argument ?
Date Sun, 23 Sep 2018 16:22:42 GMT
https://stackoverflow.com/questions/52140886/how-determine-the-number-of-workers-of-giraph-to-set-in-w-argument

Il giorno dom 2 set 2018 alle ore 22:33 Francesco Sclano <
sclano.francesco@gmail.com> ha scritto:

> I'm using an ec2 hadoop cluster that is comprised of 20 c3.8xlarge
> machines, each having 60 GB RAM and 32 virtual CPUs.
> In every machine I set up yarn and mapreduce settings as documented here
> https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html,
> i.e. as showed below:
>
> c3.8xlarge
> Configuration Option    Default Value
> mapreduce.map.java.opts    -Xmx1331m
> mapreduce.reduce.java.opts    -Xmx2662m
> mapreduce.map.memory.mb    1664
> mapreduce.reduce.memory.mb    3328
> yarn.app.mapreduce.am.resource.mb    3328
> yarn.scheduler.minimum-allocation-mb    32
> yarn.scheduler.maximum-allocation-mb    53248
> yarn.nodemanager.resource.memory-mb    53248
>
>
> Now what criteria I have to use in order to determine the most appropriate
> number of workers to use with giraph? I.e. what number I have to use for -w
> argument? Is that criteria related to above settings?
>
> --
> Francesco Sclano
>


-- 
Francesco Sclano

Mime
View raw message