hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hung (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6840) Implement zookeeper based store for scheduler configuration updates
Date Fri, 15 Sep 2017 01:20:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167196#comment-16167196
] 

Jonathan Hung edited comment on YARN-6840 at 9/15/17 1:20 AM:
--------------------------------------------------------------

Thanks [~leftnoteasy] uploaded 006 patch:
bq. Then I suggest to add this as a Javadoc to the base class's method, this should be respected
by all future implementations.
Added
bq. Avoid call CapacityScheduler methods from provider.
Done. Also exposed the ConfigurationMutationACLPolicy in MutableConfigurationProvider, since
the policy calls cs.getQueue to check queue ACLs. (ACL checking is also done in RMWebServices
just like the logging/refreshing/confirming.)
bq. Discard pending mutation during recovery. (Make sure that AdminService#refreshAll won't
include the pending mutation too).
Done. Also got rid of the transaction id logic, since there is only one pending mutation at
any time. Also added some stuff in TestZKConfigurationStore#testFailoverReadsFromUpdatedStore
to ensure pending mutation is not read on failover.
bq. MutableScheduler#updateConfiguration can be removed. Instead we need an API to get confProvider.
Done, added aclPolicy/logging/confirming API to MutableConfigurationProvider

Also since the pendingMutation/YarnConfigurationStore APIs changed, it's possible some of
the LeveldbConfigurationStore logic broke, but I will address this in YARN-7046.


was (Author: jhung):
Thanks [~leftnoteasy], 
bq. Then I suggest to add this as a Javadoc to the base class's method, this should be respected
by all future implementations.
Added
bq. Avoid call CapacityScheduler methods from provider.
Done. Also exposed the ConfigurationMutationACLPolicy in MutableConfigurationProvider, since
the policy calls cs.getQueue to check queue ACLs. (ACL checking is also done in RMWebServices
just like the logging/refreshing/confirming.)
bq. Discard pending mutation during recovery. (Make sure that AdminService#refreshAll won't
include the pending mutation too).
Done. Also got rid of the transaction id logic, since there is only one pending mutation at
any time. Also added some stuff in TestZKConfigurationStore#testFailoverReadsFromUpdatedStore
to ensure pending mutation is not read on failover.
bq. MutableScheduler#updateConfiguration can be removed. Instead we need an API to get confProvider.
Done, added aclPolicy/logging/confirming API to MutableConfigurationProvider

Also since the pendingMutation/YarnConfigurationStore APIs changed, it's possible some of
the LeveldbConfigurationStore logic broke, but I will address this in YARN-7046.

> Implement zookeeper based store for scheduler configuration updates
> -------------------------------------------------------------------
>
>                 Key: YARN-6840
>                 URL: https://issues.apache.org/jira/browse/YARN-6840
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Jonathan Hung
>         Attachments: YARN-6840-YARN-5734.001.patch, YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch,
YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, YARN-6840-YARN-5734.006.patch
>
>
> Right now there is only in-memory and leveldb based configuration store supported. Need
one which supports RM HA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message