hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuan Gong <xg...@hortonworks.com>
Subject Re: Can RM ignore heartbeats?
Date Tue, 24 Feb 2015 18:42:10 GMT
Hey, Fabio:
     In RM side, it will keep tracking on resource request status, such as how many containers
that AM has requested, how many containers that RM has assigned to AM, how many containers
that is pending, etc.
     In AM side, it is user's code. It should keep tracking on the similar resource request
status, too. If AM keeps asking the resource, eventually it will hit the queue limit/user
limit, and the resources will not be allocated.
     Also, the allocate is a blocking call. It will always get something back (could be nothing,
some of the request resources, all request resources). AM should use this information to update
the resource request status.


Xuan Gong

From: "Fabio C." <anytek88@gmail.com<mailto:anytek88@gmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 2:41 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Can RM ignore heartbeats?

Hi everyone,
I have a question about the ResourceManager behavior:
when the ResourceManager allocates a container, it takes some time before the NMToken is sent
and then received by the ApplicationMaster.
During this time, it is possible to receive another heartbeat from the AM, equal to the last
one (since the AM is not aware of the allocated resources).
Is there any policy in YARN that makes the RM aware of this and ignore this last heartbeat?
I ask this because I would expect way more superfluous containers allocated, in comparison
to the ones I can see from the logs.

Thanks in advance


View raw message