hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
Date Thu, 23 Jun 2016 21:14:16 GMT

     [ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kuhu Shukla updated YARN-4280:
------------------------------
    Attachment: YARN-4280.005.patch

Revised patch to address comments by Jason. This approach eliminates the need to store blockedResource
and instead uses childLimits.headroom when a QUEUE_SKIPPED allocation state is received, for
determining what resource limit should be passed down to the next childQueue. Also, the 'skipped'
boolean is extended in CSAssignment into an enum reflecting the skippedType which would be
OTHER_SKIPPED for legacy skipped=true, QUEUE_LIMIT_SKIPPED for our 'blocked' case, NO_SKIPPED
otherwise.

> CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
> ----------------------------------------------------------------------------------------
>
>                 Key: YARN-4280
>                 URL: https://issues.apache.org/jira/browse/YARN-4280
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>    Affects Versions: 2.6.1, 2.8.0, 2.7.1
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-4280.001.patch, YARN-4280.002.patch, YARN-4280.003.patch, YARN-4280.004.patch,
YARN-4280.005.patch
>
>
> Consider the following scenario:
> There are 2 queues A(25% of the total capacity) and B(75%), both can run at total cluster
capacity. There are 2 applications, appX that runs on Queue A, always asking for 1G containers(non-AM)
and appY runs on Queue B asking for 2 GB containers.
> The user limit is high enough for the application to reach 100% of the cluster resource.

> appX is running at total cluster capacity, full with 1G containers releasing only one
container at a time. appY comes in with a request of 2GB container but only 1 GB is free.
Ideally, since appY is in the underserved queue, it has higher priority and should reserve
for its 2 GB request. Since this request puts the alloc+reserve above total capacity of the
cluster, reservation is not made. appX comes in with a 1GB request and since 1GB is still
available, the request is allocated. 
> This can continue indefinitely causing priority inversion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message