hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7587) Skip dispatching opportunistic containers to nodes whose queue is already full
Date Fri, 01 Dec 2017 08:05:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274088#comment-16274088
] 

Weiwei Yang edited comment on YARN-7587 at 12/1/17 8:04 AM:
------------------------------------------------------------

Thanks [~kkaranasos] and [~asuresh] for the review,

bq. We can explain a bit more in comments the different max queue lengths

I have added some more comments, and changed the variable name from term {{maxQueueLength}}
to {{queueCapacity}}. Hopefully this helps to avoid confusion.

bq. Rather than performing an extra iteration to filter the list, maybe it might be better
to modify the getMetric method in the comparator to return MAX_INT / MIN_INT where appropriate
if the isQueueFull is true for the clusternode

I am not sure if that's gonna work. The objective is to exclude some unqualified nodes, if
only modify the getMetric method in comparator, it will still return all nodes. But I agree,
an extra iteration is not necessary. I have removed the filter code in v2 patch, and replaced
that by the check before adding the sorted nodes to the {{retList}}.

bq. we should do the same for de-comissioned nodes as well

Do you mean we should skip dispatching any O containers to the de-commissioned nodes? If so
I totally agree with that (I would be surprised if it is not done so). This can be tracked
in another JIRA.

Thanks.


was (Author: cheersyang):
bq. We can explain a bit more in comments the different max queue lengths

I have added some more comments, and changed the variable name from term {{maxQueueLength}}
to {{queueCapacity}}. Hopefully this helps to avoid confusion.

bq. Rather than performing an extra iteration to filter the list, maybe it might be better
to modify the getMetric method in the comparator to return MAX_INT / MIN_INT where appropriate
if the isQueueFull is true for the clusternode

I am not sure if that's gonna work. The objective is to exclude some unqualified nodes, if
only modify the getMetric method in comparator, it will still return all nodes. But I agree,
an extra iteration is not necessary. I have removed the filter code in v2 patch, and replaced
that by the check before adding the sorted nodes to the {{retList}}.

bq. we should do the same for de-comissioned nodes as well

Do you mean we should skip dispatching any O containers to the de-commissioned nodes? If so
I totally agree with that (I would be surprised if it is not done so). This can be tracked
in another JIRA.

Thanks.

> Skip dispatching opportunistic containers to nodes whose queue is already full
> ------------------------------------------------------------------------------
>
>                 Key: YARN-7587
>                 URL: https://issues.apache.org/jira/browse/YARN-7587
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.9.0
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: YARN-7587.001.patch, YARN-7587.002.patch
>
>
> When there is a lot of opportunistic container requests, even if NM queue is full, RM
continues to dispatch O containers to that node. Such allocations will be killed immediately
by NM causing a waste of resource. Propose to exclude such nodes in the load monitor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message