hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5289) Use checksum in localization requests
Date Thu, 23 Jun 2016 08:50:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346075#comment-15346075
] 

Steve Loughran commented on YARN-5289:
--------------------------------------

Note that this addresses some situations with filesystems and the localizer, where it's reporting
FS timing differences even when the files are unchanged. Checking purely on timestamp , while
simple, is a pretty brittle form of validation.

BTW, looking at the code of getFileChecksum in HDFS, it can throw {{UnsupportedFileSystemException}}
if a symlink points to an FS without the feature, whereas the API says "return null". YARN
code had better check for both when building requests; when localizing, skip checks if checksum
== 0 

> Use checksum in localization requests
> -------------------------------------
>
>                 Key: YARN-5289
>                 URL: https://issues.apache.org/jira/browse/YARN-5289
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Yesha Vora
>
> Currently, timestamps is being used in  localization requests. Because using timestamps
is always pretty brittle. ( we notice a clock jump in the VM.)
> For the filesystem supporting getFileChecksum(Path), the checksum can be used instead
of timestamps in localisation requests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message