hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6678) Committer thread crashes with IllegalStateException in async-scheduling mode of CapacityScheduler
Date Tue, 20 Jun 2017 17:15:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056108#comment-16056108
] 

Wangda Tan commented on YARN-6678:
----------------------------------

bq. It's necessary to add new method if spy on app attempt. I'll try to find another way to
test this problem, for example, spy on CapacityScheduler instance
Or you can extend CapacityScheduler and add the method in the test case, it's better to avoid
test-only method to production code. 

> Committer thread crashes with IllegalStateException in async-scheduling mode of CapacityScheduler
> -------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6678
>                 URL: https://issues.apache.org/jira/browse/YARN-6678
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.9.0, 3.0.0-alpha3
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>         Attachments: YARN-6678.001.patch, YARN-6678.002.patch
>
>
> Error log:
> {noformat}
> java.lang.IllegalStateException: Trying to reserve container container_e10_1495599791406_7129_01_001453
for application appattempt_1495599791406_7129_000001 when currently reserved container container_e10_1495599791406_7123_01_001513
on node host: node0123:45454 #containers=40 available=... used=...
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode.reserveResource(FiCaSchedulerNode.java:81)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.reserve(FiCaSchedulerApp.java:1079)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:795)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2770)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run(CapacityScheduler.java:546)
> {noformat}
> Reproduce this problem:
> 1. nm1 re-reserved app-1/container-X1 and generated reserve proposal-1
> 2. nm2 had enough resource for app-1, un-reserved app-1/container-X1 and allocated app-1/container-X2
> 3. nm1 reserved app-2/container-Y
> 4. proposal-1 was accepted but throw IllegalStateException when applying
> Currently the check code for reserve proposal in FiCaSchedulerApp#accept as follows:
> {code}
>           // Container reserved first time will be NEW, after the container
>           // accepted & confirmed, it will become RESERVED state
>           if (schedulerContainer.getRmContainer().getState()
>               == RMContainerState.RESERVED) {
>             // Set reReservation == true
>             reReservation = true;
>           } else {
>             // When reserve a resource (state == NEW is for new container,
>             // state == RUNNING is for increase container).
>             // Just check if the node is not already reserved by someone
>             if (schedulerContainer.getSchedulerNode().getReservedContainer()
>                 != null) {
>               if (LOG.isDebugEnabled()) {
>                 LOG.debug("Try to reserve a container, but the node is "
>                     + "already reserved by another container="
>                     + schedulerContainer.getSchedulerNode()
>                     .getReservedContainer().getContainerId());
>               }
>               return false;
>             }
>           }
> {code}
> The reserved container on the node of reserve proposal will be checked only for first-reserve
container.
> We should confirm that reserved container on this node is equal to re-reserve container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message