hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuan Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue
Date Wed, 10 Oct 2012 05:23:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472989#comment-13472989

Chuan Liu commented on HADOOP-8564:

Attaching a patch and update the JIRA title to reflect the change.

We port and extend Hadoop native libraries to Windows.

The POSIX native functions and flags are moved under the new nested class NativeIO.POSIX.
The Windows functions and flags are created under NativeIO.Windows.

We spent some time on investigating how to map POSIX APIs, specially all the flags to Windows
equivalent. However, this seems very difficult if even possible given all the IO options and
error codes.

Instead, we created some special IO functions in NativeIO, i.e. getShareDeleteFileInputStream(),
getCreateForWriteFileOutputStream() that abstract the IO usage pattern.

We changed the related data node functions to use the new native library functions to get
the desired I/O streams.

Some new test cases are added to TestNativeIO. TestFileConcurrentReader is fixed to test concurrent
reading and writing scenarios.
> Port and extend Hadoop native libraries for Windows to address datanode concurrent reading
and writing issue
> ------------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-8564
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8564
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 1-win
>            Reporter: Chuan Liu
>            Assignee: Chuan Liu
>         Attachments: HADOOP-8564-branch-1-win.patch
> HDFS files are made up of blocks. First, let’s look at writing. When the data is written
to datanode, an active or temporary file is created to receive packets. After the last packet
for the block is received, we will finalize the block. One step during finalization is to
rename the block file to a new directory. The relevant code can be found via the call sequence:
FSDataSet.finalizeBlockInternal -> FSDir.addBlock.
> {code} 
>         if ( ! metaData.renameTo( newmeta ) ||
>             ! src.renameTo( dest ) ) {
>           throw new IOException( "could not move files for " + b +
>                                  " from tmp to " + 
>                                  dest.getAbsolutePath() );
>         }
> {code}
> Let’s then switch to reading. On HDFS, it is expected the client can also read these
unfinished blocks. So when the read calls from client reach datanode, the datanode will open
an input stream on the unfinished block file.
> The problem comes in when the file is opened for reading while the datanode receives
last packet from client and try to rename the finished block file. This operation will succeed
on Linux, but not on Windows .  The behavior can be modified on Windows to open the file with
FILE_SHARE_DELETE flag on, i.e. sharing the delete (including renaming) permission with other
processes while opening the file. There is also a Java bug ([id 6357433|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6357433])
reported a while back on this. However, since this behavior exists for Java on Windows since
JDK 1.0, the Java developers do not want to break the backward compatibility on this behavior.
Instead, a new file system API is proposed in JDK 7.
> As outlined in the [Java forum|http://www.java.net/node/645421] by the Java developer
(kbr), there are three ways to fix the problem:
> # Use different mechanism in the application in dealing with files.
> # Create a new implementation of InputStream abstract class using Windows native code.
> # Patch JDK with a private patch that alters FileInputStream behavior.
> For the third option, it cannot fix the problem for users using Oracle JDK.
> We discussed some options for the first approach. For example one option is to use two
phase renaming, i.e. first hardlink; then remove the old hardlink when read is finished. This
option was thought to be rather pervasive.  Another option discussed is to change the HDFS
behavior on Windows by not allowing client reading unfinished blocks. However this behavior
change is thought to be problematic and may affect other application build on top of HDFS.
> For all the reasons discussed above, we will use the second approach to address the problem.
> If there are better options to fix the problem, we would also like to hear about them.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message