Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AF1F717BD8 for ; Mon, 27 Oct 2014 23:53:35 +0000 (UTC) Received: (qmail 19106 invoked by uid 500); 27 Oct 2014 23:53:35 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 19054 invoked by uid 500); 27 Oct 2014 23:53:35 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 19042 invoked by uid 99); 27 Oct 2014 23:53:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Oct 2014 23:53:35 +0000 Date: Mon, 27 Oct 2014 23:53:35 +0000 (UTC) From: "Xiaoyu Yao (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-7291) Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186062#comment-14186062 ] Xiaoyu Yao commented on HDFS-7291: ---------------------------------- Validated the V2 patch passed TestNativeIO, TestScrLazyPersistFiles and TestLazyPersistFiles under Windows Server 2012, CentOS 5.8 (LKV: 2.6.18 without sendfile support) and Ubuntu 14 (LKV: 3.13.0 with sendfile support). > Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows > ------------------------------------------------------------------------------------ > > Key: HDFS-7291 > URL: https://issues.apache.org/jira/browse/HDFS-7291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode > Affects Versions: 2.6.0 > Reporter: Xiaoyu Yao > Assignee: Xiaoyu Yao > Attachments: HDFS-7291.0.patch, HDFS-7291.1.patch, HDFS-7291.2.patch > > > HDFS-7090 changes to persist in-memory replicas using unbuffered IO on Linux and Windows. On Linux distribution, it relies on the sendfile() API between two file descriptors to achieve unbuffered IO copy. According to Linux document at http://man7.org/linux/man-pages/man2/sendfile.2.html, this is only supported on Linux kernel 2.6.33+. > As pointed by Haowei in the discussion below, FileChannel#transferTo already has support for native unbuffered IO on POSIX platform. On Windows, JDK 6/7/8 has not implemented native unbuffered IO yet. We change to use FileChannel#transfer for POSIX and our own native wrapper of CopyFileEx on Windows for unbuffered copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)