hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
Date Mon, 24 Oct 2016 05:17:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15600976#comment-15600976
] 

Wangda Tan commented on YARN-5716:
----------------------------------

Thanks [~jianhe]!

And thanks comments from [~sunilg].

For 1)/3).
>From my existing test result, the CPU time spent on write lock (which is under scheduler's
accept/apply) is only a very small proportion comparing to the CPU time spent on read lock.
It is around 3% to 10%. IAW, write lock is not a bottleneck, as of now, we cannot get a significant
perf gain from the improvements in writelock. So I'd prefer to keep them as-is to just make
it simpler.

For 2). Yeah you're correct, it is a part of my next patch, I plan to move resource-request
related functionalities from AppSchedulingInfo to SchedulingPlacementSet. I will share my
POC patch soon.

For 4). I'm not quite sure about what you meant, do you want to add comments to indicate it
should be a read-only class or you want to remove writing APIs from these classes?

For 5). I think you were talking about these logics:

{code}
    for (ContainerAllocationProposal<A,N> c : this.containersToReserve) {
      Resources.addTo(totalReservedResource,
          c.getAllocatedOrReservedResource());
      for (SchedulerContainer<A,N> r : c.getToRelease()) {
        Resources.addTo(totalReleasedResource,
            r.getRmContainer().getAllocatedOrReservedResource());
      }
    }
{code}
For each allocated/reserved container, we can have a to-release containers : {{ContainerAllocationProposal#toRelease}},
which means we will kill/release these containers to allocate/reserve the given container.
To distinguish these to-release containers from {{ResourceCommitRequest#toReleaseContainers}}.
I call them "conditional release containers" : we will only release them when trying to allocate/reserve
new resource.

And I think we need to account both of conditional/unconditional to-release containers to
the total release-able resource.

Let me know if you have any comments.

Thanks,

For example continuous-reservation-looking/lazy-preemption. 

> Add global scheduler interface definition and update CapacityScheduler to use it.
> ---------------------------------------------------------------------------------
>
>                 Key: YARN-5716
>                 URL: https://issues.apache.org/jira/browse/YARN-5716
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-5716.001.patch, YARN-5716.002.patch, YARN-5716.003.patch, YARN-5716.004.patch,
YARN-5716.005.patch, YARN-5716.006.patch, YARN-5716.007.patch
>
>
> Target of this JIRA:
> - Definition of interfaces / objects which will be used by global scheduling, this will
be shared by different schedulers.
> - Modify CapacityScheduler to use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message