hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Badger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5641) Localizer leaves behind tarballs after container is complete
Date Tue, 13 Sep 2016 15:34:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487533#comment-15487533
] 

Eric Badger commented on YARN-5641:
-----------------------------------

The problem appears to be a race between FileUtil.java:unTarUsingTar finishing and the DeletionService
deleting the localized files. If the container is killed before the untar is finished, then
it will immediately try and delete the localized directory. Since tar is running independently
of the DeletionService, it will continue to untar even after the directory contents have been
deleted. This is why we see localized files even when the DeletionService runs successfully.

I believe that the fix is to kill the untar process when the localize process is killed. If
we can't kill it, then we at least need to wait for it to finish so that we can clean up the
entire contents of what it untarred.

> Localizer leaves behind tarballs after container is complete
> ------------------------------------------------------------
>
>                 Key: YARN-5641
>                 URL: https://issues.apache.org/jira/browse/YARN-5641
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>
> The localizer sometimes fails to clean up extracted tarballs leaving large footprints
that persist on the nodes indefinitely. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message