hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bikas Saha <bi...@apache.org>
Subject RE: yarn does not allocate enough tasks/containers to my available node
Date Tue, 24 Nov 2015 22:03:57 GMT
Which scheduler is being used? Capacity/Fair/Something else?

 

From: Nicolae Marasoiu [mailto:nicolae.marasoiu@adswizz.com] 
Sent: Monday, November 23, 2015 7:59 AM
To: user@hadoop.apache.org
Subject: yarn does not allocate enough tasks/containers to my available node

 

Hi,

 

Tasks are allocated to my nodes by memory.

Initially they are allocated ok across the cluster.

After a while, one of the nodes does not receive new tasks fast enough: it
gets to 0 tasks and from time to time I see it having 1 task which it
finished in seconds.

 

It is true that I currently have a problem of many small input files. 

And probably the fact that the nodes are oversubscribed in cpu by a factor
of 2-3 (according to load average) is not helping.

 

But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node
at once (not one by one), and 2. why yarn is slow in allocating tasks? (I
understand that allocating a new task/container in a few seconds may/may not
be considered slow).

 

Pls advise,

Nicu


Mime
View raw message