hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
Date Tue, 08 Sep 2015 21:08:46 GMT

     [ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wangda Tan updated YARN-1651:
-----------------------------
    Attachment: YARN-1651-4.YARN-1197.patch

Thanks comments! [~mding].

bq. I only mention this because pullNewlyAllocatedContainers() has a check for null for the
same logic, so I think we may want to make it consistent?
Yes you're correct, updated code, thanks.

bq. So, based on my understanding, if an application has reserved some resource for a container
resource increase request on a node, that amount of resource should never be unreserved in
order for the application to allocate a regular container on some other node. But that doesn't
seem to be the case right now? Can you confirm?
Done, now added check to {{getNodeIdToUnreserve}}, will check if a container is a increase
reservation before cancel it.

bq. I think it will be desirable to implement a pendingDecrease set in SchedulerApplicationAttempt,
and corresponding logic, just like SchedulerApplicationAttempt.pendingRelease. This is to
guard against the situation when decrease requests are received while RM is in the middle
of recovery, and has not received all container statuses from NM yet.
I agree the general idea, and we should do the similar thing. However, I'm not sure caching
in RM is a good idea, potentially a malicious AM can send millions of unknown-to-be-decreased-containers
to RM when RM started. Maybe it's better to cache in AMRMClient side. I think we can do this
in a separated JIRA? Could you file a new JIRA for this if you agree?

bq. Some nits...
Addressed.

Uploaded ver.4 patch.

> CapacityScheduler side changes to support increase/decrease container resource.
> -------------------------------------------------------------------------------
>
>                 Key: YARN-1651
>                 URL: https://issues.apache.org/jira/browse/YARN-1651
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager, scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-1651-1.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch,
YARN-1651-4.YARN-1197.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message