hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6315) Improve LocalResourcesTrackerImpl#isResourcePresent to return false for corrupted files
Date Tue, 14 Mar 2017 22:21:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925152#comment-15925152
] 

Jason Lowe commented on YARN-6315:
----------------------------------

Thanks for the patch!  Looks good overall, just some minor nits:

Extra semicolon on "import java.nio.file.Files;;"

The try/catch block covers more than necessary.  Ideally it would not cover the checkLocalResource
call.

IOExceptions are treated like the resource is there.  The prior exists call that this is replacing
would return false if an exception occurred.

I think it might be useful to emit at least an info message (if not warn) indicating a resource
we thought was there is no longer there, or if it's corrupted what the size diffs are.  That
could help debugging cases where a nodemanager keeps relocalizing when it shouldn't.

static Mockito.when import is not near the other static Mockito imports


> Improve LocalResourcesTrackerImpl#isResourcePresent to return false for corrupted files
> ---------------------------------------------------------------------------------------
>
>                 Key: YARN-6315
>                 URL: https://issues.apache.org/jira/browse/YARN-6315
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.3, 2.8.1
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-6315.001.patch, YARN-6315.002.patch
>
>
> We currently check if a resource is present by making sure that the file exists locally.
There can be a case where the LocalizationTracker thinks that it has the resource if the file
exists but with size 0 or less than the "expected" size of the LocalResource. This JIRA tracks
the change to harden the isResourcePresent call to address that case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message