hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7291) Persist in-memory replicas using unbuffered IO should only applies to supported Linux version
Date Mon, 27 Oct 2014 16:49:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185367#comment-14185367

Xiaoyu Yao commented on HDFS-7291:

Thanks Haowei pointing the {code}FileChannel#transferTo(){/code} source code link. 
On supported platform, FileChannel#transferTo() tries native unbuffered API through JNI Java_sun_nio_ch_FileChannelImpl_transferTo0()
when possible while FileChannel#transferFrom() does not. This difference can only be found
from the source code and is not mentioned in JDK document.

On Linux/Solaris,  Java_sun_nio_ch_FileChannelImpl_transferTo0() calls sendfile64() to achieve
unbuffered copy. 
On Windows, Java_sun_nio_ch_FileChannelImpl_transferTo0() in OpenJDK7 simply returns *IOS_UNSUPPORTED*
without using CopyFileEX. 

As a result, we will still need native unbuffered copy wrapper for Windows but FileChannel#transferTo
should be good enough for us on Linux. In the long term, we should put a patch for OpenJDK
on Windows that implements Java_sun_nio_ch_FileChannelImpl_transferTo0 with CopyFileEx.

> Persist in-memory replicas using unbuffered IO should only applies to supported Linux
> ---------------------------------------------------------------------------------------------
>                 Key: HDFS-7291
>                 URL: https://issues.apache.org/jira/browse/HDFS-7291
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Xiaoyu Yao
>            Assignee: Xiaoyu Yao
>         Attachments: HDFS-7291.0.patch, HDFS-7291.1.patch
> HDFS-7090 changes to persist in-memory replicas using unbuffered IO on Linux and Windows.
On Linux distribution, it relies on the sendfile() API between two file descriptors to achieve
unbuffered IO copy. According to Linux document at http://man7.org/linux/man-pages/man2/sendfile.2.html,
this is only supported on Linux kernel 2.6.33+.  This JIRA is to limit the usage of sendfile()
for lazy persist only on Linux distribution with kernel version higher than 2.6.33. For unsupported
version, lazy persist will fallback to normal buffered IO.

This message was sent by Atlassian JIRA

View raw message