hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6148) NM node count reported to AM in Allocate Response should consider requested node label partitions.
Date Mon, 06 Mar 2017 10:00:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896978#comment-15896978
] 

Varun Saxena edited comment on YARN-6148 at 3/6/17 10:00 AM:
-------------------------------------------------------------

Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios
which need to be handled.
So following is the plan regarding what will be done.

# We will send a label to node count map containing label to active NM count mapping for requested
labels.
# If labels are disabled, we will report cluster count as earlier and not send anything in
label to active NM count mapping. AM can consider only the cluster node count reported earlier
in such cases. This will also help us in handling the case for rolling downgrades.
# Non Exclusive labels will not be reported in the label to node count unless explicitly requested
by AM.
# Mapreduce AM would consider both map and reduce label expression separately, if they are
different, while deciding on ignoring AM blacklisting. We will check label to node count mapping
to determine the applicable node counts for both map and reduce. 
# Queue can have a default label too. This needs to be reported to AM as well. Also needs
to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and
AM currently does not know anything about default label of queue.
# Also we can report AM node label from Application Submission context in Register AM response.
This is because RM can consider label in ASC if none is specified in AM ask.
#  Container info sent while reporting allocated containers would also contain label on which
container was allocated on. Based on this info we will determine if the node assigned is a
default label of queue, map/reduce partition or non exclusive label. We will not count non
exclusive label towards ignoring blacklisting. 

Thoughts?


was (Author: varun_saxena):
Offline, we (me, [~Naganarasimha] and [~bibinchundatt]) had discussed some more scenarios
which need to be handled.
So following is the plan regarding what will be done.

# We will send a label to node count map containing label to active NM count mapping for requested
labels.
# If labels are disabled, we will report cluster count as earlier and not send anything in
label to active NM count mapping. AM can consider only the cluster node count reported earlier
in such cases. This will also help us in handling the case for rolling downgrades.
# Non Exclusive labels will not be reported in the label to node count unless explicitly requested
by AM.
# Mapreduce AM would consider both map and reduce label expression separately, if they are
different, while deciding on ignoring AM blacklisting. We will check label to node count mapping
to determine the applicable node counts for both map and reduce. 
# Queue can have a default label too. This needs to be reported to AM as well. Also needs
to be reported on move. Can be handled in YARN-6209. Active NM count may depend on it and
AM currently does not know anything about default label of queue.
#  Container info sent while reporting allocated containers would also contain label on which
container was allocated on. Based on this info we will determine if the node assigned is a
default label of queue, map/reduce partition or non exclusive label. We will not count non
exclusive label towards ignoring blacklisting. 

Thoughts?

> NM node count reported to AM in Allocate Response should consider requested node label
partitions.
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6148
>                 URL: https://issues.apache.org/jira/browse/YARN-6148
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-6148.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message