hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prakash (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1036) Distributed Cache gives inconsistent result if cache files get deleted from task tracker
Date Wed, 07 Aug 2013 20:59:48 GMT

    [ https://issues.apache.org/jira/browse/YARN-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732731#comment-13732731
] 

Ravi Prakash commented on YARN-1036:
------------------------------------

[~ojoshi] 
bq. (here we don't want to check whether file is present on disc or not.. do we have to? 
I agree that the very first time the file is localized, checking for the existence of the
file all over again is redundant. The file could go missing right after checking, and we would
be none the wiser. However in 0.23, I am willing to check again. That's why I am not proposing
the change to trunk. There's a lot of changes between trunk/branch-2 and branch-0.23 and we
are only cherry-picking those necessary for maintaining the line. That is why we are also
not backporting all the cool stuff you have done e.g. in YARN-539 and YARN-467 etc.

bq. now we should definitely check if the file is already present on disc or not. If not remove
it from local cache (DataStructure-- by delete the file I wanted to say from local cache not
from disc as it is already gone from there).
Isn't that exactly what the new code in the patch is doing?
{code}
      if (rsrc != null && (!isResourcePresent(rsrc))) {
        LOG.info("Resource " + rsrc.getLocalPath()
            + " is missing, localizing it again");
        localrsrc.remove(req);
        rsrc = null;
      }
{code}
                
> Distributed Cache gives inconsistent result if cache files get deleted from task tracker

> -----------------------------------------------------------------------------------------
>
>                 Key: YARN-1036
>                 URL: https://issues.apache.org/jira/browse/YARN-1036
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 0.23.9
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>         Attachments: YARN-1036.branch-0.23.patch
>
>
> This is a JIRA to backport MAPREDUCE-4342. I had to open a new JIRA because that one
had been closed. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message