hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: Containers of required size are not being allocated.
Date Tue, 04 Nov 2014 06:02:17 GMT

I bet you are not setting different Resource-Request-priorities on your requests.

It is a current limitation (https://issues.apache.org/jira/browse/YARN-314) of not being able
to support resources of different sizes against a single priority.


On Nov 3, 2014, at 1:23 AM, Smita Deshpande <smita.deshpande@cumulus-systems.com> wrote:

> Hi All,
> I have a single YARN application running on a cluster of 1 master node(Resourcemanager)
and 3 slave nodes (Nodemanagers) - total memory= 24GB, total vcores= 12. I am using Hadoop
2.4.1 and scheduler is Capacity-Scheduler with DominantResourceCalculator. This application
submits 10 container requests to ResourceManager through the asynchronous AMRM client ( AMRMClientAsync).
The behavior I observed was that the container request asking for the most resources gets
allocated first. For eg. 9 requests are for 2GB containers and 1 request is for a 4 GB container,
then the 4GB container gets allocated and here's where it runs into a problem - It just stops
there - it doesn't allocate any containers for all the 2 GB requests!
> Once a container is allocated my app runs a task on it and once it is finished , app
submits a fresh container request for this task (the tasks are of a repeating nature). Then
again the RM allocates a 4GB container for this request, completely ignoring all the requests
in between. I don't know if it is an issue with the AMRMClientAsync class- I haven't tried
using the normal AMRMclient yet. Now I have tasks that require 2 GB and tasks that require
4 GB memory. Because of this problem it is only running the 4 GB task again and again.
> If all the container requests are identical in terms of resources, then this problem
disappears and RM allocates all requested containers.
> Couldn't find any documentation/known bugs relating to this. I've hit a brick wall here,
any help would be greatly appreciated. Thanks in advance
> -SMita

View raw message