hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6315) Improve LocalResourcesTrackerImpl#isResourcePresent to return false for corrupted files
Date Fri, 17 Mar 2017 15:58:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930205#comment-15930205

Jason Lowe commented on YARN-6315:

I tried to run this in an end-to-end test and found it doesn't work in practice.  I was under
the mistaken impression that the size specified in the LocalResourceRequest was used to verify
the correct file was being localized, but that's not the case.  It only uses the _timestamp_
to verify the correct version of the file is being downloaded.  The size is ignored.  In my
case the request actually contained the value -1 for the size, so it always thought the size
mismatched and would re-localize the file.  That's not good.

I thought we could pivot from the (now untrustworthy) size in the request to the size in the
LocalizedResource. That's a value the NM computes directly during localization, so that will
be correct.  However this is the size of the entire directory containing the localized resource
(whether that's a file, archive, or directory), so it includes extra things like the .crc
file from LocalFileSystem, etc.  In order to match the sizes we'd have to do the same logic
being done by the localizer which is a DU of the directory.  That's going to be too expensive
to do for every local resource lookup by a container launch.

> Improve LocalResourcesTrackerImpl#isResourcePresent to return false for corrupted files
> ---------------------------------------------------------------------------------------
>                 Key: YARN-6315
>                 URL: https://issues.apache.org/jira/browse/YARN-6315
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.3, 2.8.1
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-6315.001.patch, YARN-6315.002.patch, YARN-6315.003.patch, YARN-6315.004.patch
> We currently check if a resource is present by making sure that the file exists locally.
There can be a case where the LocalizationTracker thinks that it has the resource if the file
exists but with size 0 or less than the "expected" size of the LocalResource. This JIRA tracks
the change to harden the isResourcePresent call to address that case.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message