hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Young Chen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
Date Wed, 21 Feb 2018 00:25:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370783#comment-16370783
] 

Young Chen edited comment on YARN-7708 at 2/21/18 12:24 AM:
------------------------------------------------------------

The rationale for this policy is to lower load on sub clusters that are receiving an abnormal
amount of jobs or are unable to keep up for performance reasons. To alleviate job failures/excessive
delays, this policy attempts to redirect load away from highly loaded sub clusters by updating
a WeightedLocalityPolicyManager with modified weights.

This patch introduces the LoadBasedGlobalPolicy. It's configurable in the following ways:
 * MAX_EDIT: The maximum number of sub clusters weights this policy will edit. The top N clusters
by pending load will have their policy weights scaled down.
 * MIN_PENDING: The minimum number of pending applications before a sub cluster weight qualifies
for editing.
 * MAX_PENDING: The maximum number of pending applications in the scalable range. Sub clusters
exceeding this will have their policy weights set to MIN_WEIGHT
 * MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster.
 * SCALING: The scaling method that maps pending load to sub cluster policy weight. Currently
there are three scaling methods: quadratic, log, and linear. 


was (Author: youchen):
The rationale for this policy is to lower load on sub clusters that are receiving an abnormal
amount of jobs or are unable to keep up for performance reasons. To alleviate job failures/excessive
delays, this policy attempts to redirect load away from highly loaded sub clusters by updating
a WeightedLocalityPolicyManager with modified weights.

This patch introduces the LoadBasedGlobalPolicy. It's configurable in the following ways:
 * MAX_EDIT: The maximum number of sub clusters this policy will throttle. The top N clusters
by pending load will have their policy weights scaled down.
 * MIN_PENDING: The minimum number of pending applications before a sub cluster weight qualifies
for editing.
 * MAX_PENDING: The maximum number of pending applications in the scalable range. Sub clusters
exceeding this will have their policy weights set to MIN_WEIGHT
 * MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster.
 * SCALING: The scaling method that maps pending load to sub cluster policy weight. Currently
there are three scaling methods: quadratic, log, and linear. 

> [GPG] Load based policy generator
> ---------------------------------
>
>                 Key: YARN-7708
>                 URL: https://issues.apache.org/jira/browse/YARN-7708
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Carlo Curino
>            Assignee: Young Chen
>            Priority: Major
>         Attachments: YARN-7708-YARN-7402.01.cumulative.patch
>
>
> This policy reads load from the "pendingQueueLength" metrics and provides scaling into
a set of weights that influence the AMRMProxy and Router behaviors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message