hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6319) race condition between deleting app dir and deleting container dir
Date Thu, 16 Mar 2017 22:12:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929009#comment-15929009

Haibo Chen commented on YARN-6319:

Thanks [~zhiguohong] for more explanation on option 2. While I agree with you that post-callback
can completely avoid race condition, linearizing container cleanup and app cleanup will unnecessarily
slow down the application state transition process which other tasks, such as log aggregation,
depend on. Especially when you have a lot of containers for a given application, previously
the app dir cleanup task can be running concurrently with all container cleanup tasks, now
it will need to wait for all container cleanup tasks to finish. The point I wan to make is
that the race condition is safe to have as long as we ignore the fileNotException error during
deletion. I notice YARN-2902 add the code to ignore FileNotExistent error code for LCE. Is
it included in the version where you ran into this issue?

> race condition between deleting app dir and deleting container dir
> ------------------------------------------------------------------
>                 Key: YARN-6319
>                 URL: https://issues.apache.org/jira/browse/YARN-6319
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Hong Zhiguo
>            Assignee: Hong Zhiguo
> Last container (on one node) of one app complete
>     |    --> triggers async deletion of container dir (container cleanup)
>     |    --> triggers async deletion of app dir (app cleanup)
> For LCE, deletion is done by container-executor. The "app cleanup" lists sub-dir (step
1), and then unlink items one by one(step 2).   If a file is deleted by "container cleanup"
between step 1 and step2, it'll report below error and breaks the deletion.
> {code}
> ContainerExecutor: Couldn't delete file $LOCAL/usercache/$USER/appcache/application_1481785469354_353539/container_1481785469354_353539_01_000028/$FILE
- No such file or directory
> {code}
> This app dir then escape the cleanup. And that's why we always have many app dirs left
> solution 1: just ignore the error without breaking in container-executor.c::delete_path()
> solution 2: use a lock to serialize the cleanup of same app dir.
> solution 3: backoff and retry on error
> Comments are welcome.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message