hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Yang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-9413) Queue resource leak after app fail for CapacityScheduler
Date Sun, 07 Apr 2019 02:01:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tao Yang updated YARN-9413:
---------------------------
    Attachment: YARN-9413.branch-3.0.001.patch

> Queue resource leak after app fail for CapacityScheduler
> --------------------------------------------------------
>
>                 Key: YARN-9413
>                 URL: https://issues.apache.org/jira/browse/YARN-9413
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 3.1.2
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>            Priority: Major
>             Fix For: 3.3.0, 3.2.1, 3.1.3
>
>         Attachments: YARN-9413.001.patch, YARN-9413.002.patch, YARN-9413.003.patch, YARN-9413.branch-3.0.001.patch,
image-2019-03-29-10-47-47-953.png
>
>
> To reproduce this problem:
>  # Submit an app which is configured to keep containers across app attempts and should
fail after AM finished at first time (am-max-attempts=1).
>  # App is started with 2 containers running on NM1 node.
>  # Fail the AM of the application with PREEMPTED exit status which should not count
towards max attempt retry but app will fail immediately.
>  # Used resource of this queue leaks after app fail.
> The root cause is the inconsistency of handling app attempt failure between RMAppAttemptImpl$BaseFinalTransition#transition
and RMAppImpl$AttemptFailedTransition#transition:
>  # After app fail, RMAppFailedAttemptEvent will be sent in RMAppAttemptImpl$BaseFinalTransition#transition,
if exit status of AM container is PREEMPTED/ABORTED/DISKS_FAILED/KILLED_BY_RESOURCEMANAGER,
it will not count towards max attempt retry, so that it will send AppAttemptRemovedSchedulerEvent
with keepContainersAcrossAppAttempts=true and RMAppFailedAttemptEvent with transferStateFromPreviousAttempt=true.
>  # RMAppImpl$AttemptFailedTransition#transition handle RMAppFailedAttemptEvent and will
fail the app if its max app attempts is 1.
>  # CapacityScheduler handles AppAttemptRemovedSchedulerEvent in CapcityScheduler#doneApplicationAttempt,
it will skip killing and calling completion process for containers belong to this app, so
that queue resource leak happens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message