hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5261) Lease/Reclaim Extension to Yarn
Date Thu, 07 Jul 2016 10:19:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365914#comment-15365914

Yu commented on YARN-5261:

Yarn-5261 main purpose is: lease the unused resource on some nodes which already having some
important services (outside of Yarn), and make the lease/reclaim controllable by the nodes'
owners (landlords).
It also uses zookeeper to make the lease/reclaim operation recoverable even there's RM restarting
or failover, so it's very stable.

I checked the issues you mentioned, and I got:
	* Yarn-5215:
		It has similar purpose as my issue: leverage the unused resources on some nodes.
		But, it's another approach, and can get different result.

		It assumes all nodes are schedulable by Yarn, and changes the unallocated resource by reducing
the external utilization on each node dynamically.  
		My issue (Yarn-5261) is different:
		1. don't assume all nodes are schedulable (though Yarn NM is running on all nodes).  The
nodes are schedulable only when the landlord leases them.  So, it's more controllable.
		2. the total schedulable resource is estimated by the landlord and set when node is leased,
which is steadier than changing after calculating the utilization.  And, landlord can plan
how to use the resources better, not rely on automatic calculating.
		3. when the schedulable resource is over used, the containers will be killed to make sure
obeying the landlord.

	* Yarn-1011:
		It's more like a Yarn scheduler enhancement for containers, but don't consider the external
utilization at all.
		It's not proper for the scenario that running yarn on nodes with other services.
	* Yarn-5202:
		It's about resource overcommit, not the same purpose.

So, they look similar, but are different.  Users can use them based on their requirements.

> Lease/Reclaim Extension to Yarn
> -------------------------------
>                 Key: YARN-5261
>                 URL: https://issues.apache.org/jira/browse/YARN-5261
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Yu
>         Attachments: YARN-5261-1.patch, Yarn-5261.pdf
> In some clusters outside of Yarn, the machines resources are not fully utilized, e.g.,
resource usage is quite low at night.  
> To better utilize the resources while keep the existing SLA of the cluster, Lease/Reclaim
Extension to Yarn is introduced.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message