hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Trezzo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7001) If shared cache upload is terminated in the middle, the temp file will never be deleted
Date Sun, 01 Oct 2017 23:36:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187583#comment-16187583

Chris Trezzo commented on YARN-7001:

Looking at this patch, I am not entirely sure if this fixes the issue. I am thinking about
these two scenarios:
# If {{uploadFile}} returns false: {{FileUtil.copy}} has returned false. If we look into that
method, I think the only way it will return false is if the file has not been created yet,
since we pass in deleteSource as false. In this case, we do not need a deleteTempFile call.
# If {{uploadFile}} throws an IOException: Here we might have an issue. If copy throws an
IOException after it created the tmp file, but before it finished writing it, we may be stranding
tmp files. It seems like we would want a try/catch around the uploadFile. If we get an IOException,
we would want to delete the tmp file if it exists.

In reality, we could be stranding tmp files if the node manager fails at any point between
the file creation in uploadFile and the file rename later in the method. In practice, this
doesn't seem to be an issue because the time between those points is small. Maybe we could
add a try/finally around those two points where we attempt to delete the tmp file in the finally?
That at least covers the case where there is an unexpected exception.

Let me know if you think I have missed something. Thanks!

> If shared cache upload is terminated in the middle, the temp file will never be deleted
> ---------------------------------------------------------------------------------------
>                 Key: YARN-7001
>                 URL: https://issues.apache.org/jira/browse/YARN-7001
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Miklos Szegedi
>            Assignee: Sen Zhao
>         Attachments: YARN-7001.001.patch, YARN-7001.002.patch, YARN-7001.003.patch, YARN-7001.004.patch
> There is a missing deleteTempFile(tempPath);
> {code}
>       tempPath = new Path(directoryPath, getTemporaryFileName(actualPath));
>       if (!uploadFile(actualPath, tempPath)) {
>         LOG.warn("Could not copy the file to the shared cache at " + tempPath);
>         return false;
>       }
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message