hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: When does the RM ever send out node map updates?
Date Mon, 19 Oct 2015 11:04:14 GMT
>>> how does the AM determine the map of nodes.
IIUC, it is not possible now. Probably a bug which was missed till now!! 
OR 
Does all the applications need to know all the nodes updates in a cluster? It could be 1000
nodes , 10K nodes. 

And  how about when node label is configured.?


Thanks & Regards
Rohith Sharma K S

-----Original Message-----
From: Steve Loughran [mailto:stevel@hortonworks.com] 
Sent: 19 October 2015 15:01
To: yarn-dev@hadoop.apache.org
Subject: Re: When does the RM ever send out node map updates?


Thanks for that snippet.

Which brings me back to a key question: how does the AM determine the map of nodes.

And before someone says "the yarn client lets you do that", bear in mind this is for a long-lived
restartable AM; what may have been valid when the client launched it 3 weeks earlier may be
invalid on restart. 

If it were in the RegisterApplicationMasterResponse I'd pick it up there & wait for changes
—but you don't get a nodemap then

Any suggestions?

> On 19 Oct 2015, at 06:25, Rohith Sharma K S <rohithsharmaks@huawei.com> wrote:
> 
> NM node updates are sent to ApplicationMaster only when NodeManager is added/removed/'change
in NM state i.e Usable/Unusable'. Otherwise no updates are sent to AM  even at least one time
in application life time!!
> 
> I believe it is implemented purposefully to intimate AM about only unusable nodes. 
> 
> Thanks & Regards
> Rohith Sharma K S
> 
> -----Original Message-----
> From: Steve Loughran [mailto:stevel@hortonworks.com] 
> Sent: 18 October 2015 23:48
> To: yarn-dev@hadoop.apache.org
> Subject: When does the RM ever send out node map updates?
> 
> 
> I've been modifying the slider AM to wait for NM updates of node maps -but I'm not seeing
any onNodesUpdated() callbacks, at least for an AM which doesn't instantiate any containers.
> 
> When/how do those updates come?
> 

Mime
View raw message