hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Mitic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12550) NativeIO#renameTo on Windows cannot replace an existing file at the destination.
Date Wed, 04 Nov 2015 23:02:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990649#comment-14990649
] 

Ivan Mitic commented on HADOOP-12550:
-------------------------------------

Hi [~cnauroth],
Interesting problem!

I reviewed docs for Linux rename and Windows MoveFileEx with the MOVEFILE_REPLACE_EXISTING
flag, and indeed they seem compatible (noticed only one semantic difference when the target
folder already exists and it's empty). Seems to me like the right fix. I am +1 on the patch.
One minor comment, please include an inline code comment providing the background on MOVEFILE_REPLACE_EXISTING
flag.

Before you commit, would you be able to validate that existing Common/Hdfs/MR/Yarn tests did
not regress on Windows?

> NativeIO#renameTo on Windows cannot replace an existing file at the destination.
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-12550
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12550
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>         Environment: Windows
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-12550.001.patch
>
>
> {{NativeIO#renameTo}} currently has different semantics on Linux vs. Windows if a file
already exists at the destination.  On Linux, it's a passthrough to the [rename|http://linux.die.net/man/2/rename]
syscall, which will replace an existing file at the destination.  On Windows, it's a passthrough
to [MoveFile|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365239%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396],
which cannot replace an existing file at the destination and instead triggers an error.  The
easiest way to observe this difference is to run the HDFS test {{TestRollingUpgrade#testRollback}}.
 This fails on Windows due to a block recovery after truncate trying to replace a block at
an existing destination path.  This issue proposes to use [MoveFileEx|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240(v=vs.85).aspx]
on Windows with the {{MOVEFILE_REPLACE_EXISTING}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message