hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6315) Improve LocalResourcesTrackerImpl#isResourcePresent to return false for corrupted files
Date Tue, 20 Jun 2017 14:31:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055849#comment-16055849
] 

Kuhu Shukla commented on YARN-6315:
-----------------------------------

Adding an 'actualSize' for LocalizedResource and then checking it against the file attributes
in isResourcePresent() covers a subset of corruption scenarios. That is if the file size changes
after its successful download. I am leaning towards adding actualSize to reflect the "hdfs
resource size" and compare that with the local file size. This will cover any corruption caused
during download. Special case here would be directories and archives.

> Improve LocalResourcesTrackerImpl#isResourcePresent to return false for corrupted files
> ---------------------------------------------------------------------------------------
>
>                 Key: YARN-6315
>                 URL: https://issues.apache.org/jira/browse/YARN-6315
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.3, 2.8.1
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-6315.001.patch, YARN-6315.002.patch, YARN-6315.003.patch, YARN-6315.004.patch
>
>
> We currently check if a resource is present by making sure that the file exists locally.
There can be a case where the LocalizationTracker thinks that it has the resource if the file
exists but with size 0 or less than the "expected" size of the LocalResource. This JIRA tracks
the change to harden the isResourcePresent call to address that case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message