hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kawa <kawa.a...@gmail.com>
Subject Re: The number of simultaneous map tasks is unexpected.
Date Thu, 10 Jul 2014 13:31:29 GMT
> yarn.nodemanager.resource.memory-mb = 2370 MiB,
> yarn.nodemanager.resource.cpu-vcores = 2,

So, you cannot run more than 8 containers on your setup (according to your
settings, each container consumes 1GB and 1 vcore).

> Considering that I have 8 cores in my cluster and not 16 as I thought at
the beginning, starting more than 7 map tasks (and AM) is not supposed to
give me performance gains as all the cores have been used already. Am I

Right. But you can increase yarn.nodemanager.resource.cpu-vcores to 3 and
decrease the container sizes (and heap) of map and reduce tasks, and you
will be able to run more containers on the NodeManagers (assuming that you
will not overload the machine).

View raw message