hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: Resource Manager's container allocation behavior !
Date Mon, 24 Nov 2014 07:28:08 GMT
Hi Hamza Zafar

     I would like to let you know first that ApplicationMasterProtocol# allocate() has not
only for requesting container but also doubles up as a heartbeat to let the ResourceManager
know that the ApplicationMaster is alive
So basically your ApplicationMaster should be keep sending heartbeat to RM via allocate()

Container allocation will happen when NodeMager sends heartbeats to RM. This is the reason
for you allocation time reduced when  you decrease heartbet-interval-ms.

Why the application is not provided with all requested containers in first allocate call?
>> For the first call, RM updates request but allocation will happen when NM heartbeat
to RM. So for 2nd call , containers will be received by AM.

Thanks & Regards
Rohith Sharma K S

From: Hamza Zafar [mailto:11bscshzafar@seecs.edu.pk]
Sent: 22 November 2014 00:45
To: user@hadoop.apache.org
Subject: Resource Manager's container allocation behavior !

My Hadoop Cluster has 52GB memory , 56 virtual cores

Scenario: I submit an application to a default queue while there is no other application running
on the cluster. I create a request for 32 containers with same priority, 512MB memory and
1 virtual core . In the first allocate call I receive 0 containers from RM, in further allocate
calls I start receiving containers. I keep on sending allocate calls until all the containers
have been allocated.

Why the application is not provided with all requested containers in first allocate call?

I changed the configuration property "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms"
from 1000ms to 100ms .Now at 100ms heatbeat interval the container allocation time has reduced,
but still the AM has to make the same number of allocate calls as it was done before when
the heartbeat interval was 1000ms.
View raw message