hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5324) Stateless router policies implementation
Date Fri, 16 Sep 2016 19:10:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497124#comment-15497124
] 

Carlo Curino commented on YARN-5324:
------------------------------------


[~subru], thanks for the good review. I think I have addressed all your concerns. Below I
answer your questions, or explain some of the choices made in addressing your comments. 

Regarding {{SubClusterIdInfo}}
 # Done 
 
Regarding {{WeightedPolicyInfo}}
 # I addressed all comments with one exception: marshaller and unmarshallers cannot be preinitialized,
but {{JSONJAXBContext}} can.
 
Regarding {{BaseWeightedRouterPolicy}}:
 # the policyInfo check is correct I think (if the current info is not null and is the same
as the new one than nothing should be reinitialized)
 # for a router policy we don't care about what happens for amrmproxy weights.
 # I moved the checks in a validate method, but I think it is a bit redundant at the moment
(the only class that would use it is {{LoadBasedRaouterPolicy}} which it doesn't because it
would be clumsy).
 # I don't think adding {{selectSubCluster}} changes things much. The general policy API we
are implementing is pretty much the same I think. 
 # check for active subclusters is indeed somewhat repeated, though some policies (e.g, {{UniformRandomRouterPolicy}})
use it in different ways.
 
Regarding {{LoadBasedRaouterPolicy}}
 # I think we should repeat the available memory check every time as this might change based
on the latest set of info the FederationStateStoreFacade is returning for the activeSubclusters
  
Regarding {{OrderRouterPolicy}}
  # Agreed to rename this to {{PriorityRouterPolicy}}. This simple policy is very good for
testing (no randomness), and can also provide simple failover type of semantics "I want jobs
in this queue to run on subcluster 1, and if that is not available then use subcluster 2,
etc.."
 
 Regarding "Common across tests":
 # In all/most tests the set of "activeSubclusters" is chosen to be a subset of the one specified
in the policy weights. All policies are basically stateless, previous decisions should not
affect following ones so the multi invocation tests are only relevant if we check statistical
properties (e.g., like we do in {{testClusterChosenWithRightProbability}})
 # Factored out all common parts, good advice (cut 10kb and increased coverage).
 # Some of the method in FederationPoliciesTestUtil are used by the upcoming patches for AMRMProxy
(I was trying to avoid editing that class over and over at every patch). 

> Stateless router policies implementation
> ----------------------------------------
>
>                 Key: YARN-5324
>                 URL: https://issues.apache.org/jira/browse/YARN-5324
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>    Affects Versions: YARN-2915
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5324-YARN-2915.06.patch, YARN-5324-YARN-2915.07.patch, YARN-5324-YARN-2915.08.patch,
YARN-5324-YARN-2915.09.patch, YARN-5324-YARN-2915.10.patch, YARN-5324.01.patch, YARN-5324.02.patch,
YARN-5324.03.patch, YARN-5324.04.patch, YARN-5324.05.patch
>
>
> These are policies at the Router that do not require maintaing state across choices (e.g.,
weighted random).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message