hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
Date Mon, 13 Feb 2017 08:56:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15863347#comment-15863347
] 

Sunil G commented on YARN-6148:
-------------------------------

[~varun_saxena]

I have some doubts over here. 
{{allocateResponse.setNumClusterNodes(allocation.getApplicableNMCount());}}

Still we sent one integer value of node count to AM. This count is better than earlier because
its the node count for all used partitions for this app (instead of all nodes in cluster).
But I am wary about few use cases.

Lets consider  below scenario for MR:
- AM container is running in LabelA(it has 10 nodes)
- Maps and reducers are running in LabelB (10 nodes)

4 tasks are failed in one node of LabelB. Now that node ll be blacklisted. 
As per your code, node count is 20 (Am container's LabelA, Map/Reducer containers LabelB).
But for task failure, we need not have to worry about LabelA (AM container node label). We
should consider only LabelB here.

So I think we must send per label node count to MR (only for request partitions). {{<<labelA,
10>, <labelB, 10>>}} It can be a new parameter in the existing protocol. Also
MR code to be changed as per this. This may be more complete.


> NM node count reported to AM in Allocate Response should consider requested node label
partitions.
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6148
>                 URL: https://issues.apache.org/jira/browse/YARN-6148
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-6148.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message