hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1543) Stateless multiple resource manager
Date Fri, 27 Dec 2013 06:22:52 GMT

     [ https://issues.apache.org/jira/browse/YARN-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsuyoshi OZAWA updated YARN-1543:
---------------------------------

    Description: 
Currently, YARN is implemented based on statefull single resource manager architecture. This
approach can raise operation costs compare to stateless multi resource manager architecture,
because we need to care which resource manager is active to ensure availability. 

We can avoid this problem by saving all states into updatable and transactional RMStateStore
such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched
at the same time. As a result, operation cost can be reduced, because we don't need to maintain
specific(current master) resource manager.

This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute
clusters”[1].

[1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf


  was:
Currently, YARN is implemented based on statefull single resource manager architecture. This
approach can raise operation costs compare to stateless multi resource manager architecture,
because we need to care which resource manager is active to ensure availability. 

We can avoid this problem by saving all states into updatable and transactional RMStateStore
such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched
at the same time, we can solve this problem. Operation cost can be reduced, because we don't
need to maintain specific(current master) resource manager.

This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute
clusters”[1].

[1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf



> Stateless multiple resource manager
> -----------------------------------
>
>                 Key: YARN-1543
>                 URL: https://issues.apache.org/jira/browse/YARN-1543
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Tsuyoshi OZAWA
>
> Currently, YARN is implemented based on statefull single resource manager architecture.
This approach can raise operation costs compare to stateless multi resource manager architecture,
because we need to care which resource manager is active to ensure availability. 
> We can avoid this problem by saving all states into updatable and transactional RMStateStore
such as ZooKeeper or RDBMS. If we take this approach, multiple resource managers can be launched
at the same time. As a result, operation cost can be reduced, because we don't need to maintain
specific(current master) resource manager.
> This idea is based on a paper: “Omega: flexible, scalable schedulers for large compute
clusters”[1].
> [1] http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message