hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2902) Killing a container that is localizing can orphan resources in the DOWNLOADING state
Date Tue, 27 Oct 2015 20:19:28 GMT

    [ https://issues.apache.org/jira/browse/YARN-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14977077#comment-14977077
] 

Jason Lowe commented on YARN-2902:
----------------------------------

bq. I think in LocalResourcesTrackerImpl#handle, after handling RELEASE event, we should check
if the reference count is 0 and whether state of resource is DOWNLOADING. And if this is so,
call LocalResourcesTrackerImpl#removeResource.

Agreed.  We can automatically remove the resource if the refcount of a downloaded resource
ever goes to zero.  And if there's a race where another container is just trying to reference
that resource just as we're releasing (and removing) it from a killed container then either
we'll keep it because the refcount is nonzero (request comes before release) or we'll create
a new resource to track the subsequent request (release comes before request).


> Killing a container that is localizing can orphan resources in the DOWNLOADING state
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-2902
>                 URL: https://issues.apache.org/jira/browse/YARN-2902
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 2.5.0
>            Reporter: Jason Lowe
>            Assignee: Varun Saxena
>         Attachments: YARN-2902.002.patch, YARN-2902.03.patch, YARN-2902.04.patch, YARN-2902.05.patch,
YARN-2902.06.patch, YARN-2902.07.patch, YARN-2902.08.patch, YARN-2902.patch
>
>
> If a container is in the process of localizing when it is stopped/killed then resources
are left in the DOWNLOADING state.  If no other container comes along and requests these resources
they linger around with no reference counts but aren't cleaned up during normal cache cleanup
scans since it will never delete resources in the DOWNLOADING state even if their reference
count is zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message