hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Smita Deshpande <smita.deshpa...@cumulus-systems.com>
Subject Containers of required size are not being allocated.
Date Mon, 03 Nov 2014 09:23:02 GMT
Hi All,

I have a single YARN application running on a cluster of 1 master node(Resourcemanager) and
3 slave nodes (Nodemanagers) - total memory= 24GB, total vcores= 12. I am using Hadoop 2.4.1
and scheduler is Capacity-Scheduler with DominantResourceCalculator. This application submits
10 container requests to ResourceManager through the asynchronous AMRM client ( AMRMClientAsync).
The behavior I observed was that the container request asking for the most resources gets
allocated first. For eg. 9 requests are for 2GB containers and 1 request is for a 4 GB container,
then the 4GB container gets allocated and here's where it runs into a problem - It just stops
there - it doesn't allocate any containers for all the 2 GB requests!

Once a container is allocated my app runs a task on it and once it is finished , app submits
a fresh container request for this task (the tasks are of a repeating nature). Then again
the RM allocates a 4GB container for this request, completely ignoring all the requests in
between. I don't know if it is an issue with the AMRMClientAsync class- I haven't tried using
the normal AMRMclient yet. Now I have tasks that require 2 GB and tasks that require 4 GB
memory. Because of this problem it is only running the 4 GB task again and again.

If all the container requests are identical in terms of resources, then this problem disappears
and RM allocates all requested containers.

Couldn't find any documentation/known bugs relating to this. I've hit a brick wall here, any
help would be greatly appreciated. Thanks in advance

View raw message