hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolae Marasoiu <nicolae.maras...@adswizz.com>
Subject yarn does not allocate enough tasks/containers to my available node
Date Mon, 23 Nov 2015 15:58:43 GMT

Tasks are allocated to my nodes by memory.

Initially they are allocated ok across the cluster.

After a while, one of the nodes does not receive new tasks fast enough: it gets to 0 tasks
and from time to time I see it having 1 task which it finished in seconds.

It is true that I currently have a problem of many small input files.

And probably the fact that the nodes are oversubscribed in cpu by a factor of 2-3 (according
to load average) is not helping.

But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node at once (not
one by one), and 2. why yarn is slow in allocating tasks? (I understand that allocating a
new task/container in a few seconds may/may not be considered slow).

Pls advise,


View raw message