Return-Path: <hdfs-issues-return-163725-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 8596C200B3C
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Wed, 13 Jul 2016 09:39:22 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id 8422D160A81; Wed, 13 Jul 2016 07:39:22 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id CD09A160A57
	for <archive-asf-public@cust-asf.ponee.io>; Wed, 13 Jul 2016 09:39:21 +0200 (CEST)
Received: (qmail 91396 invoked by uid 500); 13 Jul 2016 07:39:21 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 91341 invoked by uid 99); 13 Jul 2016 07:39:20 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jul 2016 07:39:20 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id C94EC2C02A7
	for <hdfs-issues@hadoop.apache.org>; Wed, 13 Jul 2016 07:39:20 +0000 (UTC)
Date: Wed, 13 Jul 2016 07:39:20 +0000 (UTC)
From: "Ravikumar (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12545295.1331023614000.13377.1468395560821@Atlassian.JIRA>
In-Reply-To: <JIRA.12545295.1331023614000@Atlassian.JIRA>
References: <JIRA.12545295.1331023614000@Atlassian.JIRA> <JIRA.12545295.1331023614685@arcas>
Subject: [jira] [Commented] (HDFS-3051) A zero-copy ScatterGatherRead api
 from FSDataInputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Wed, 13 Jul 2016 07:39:22 -0000


    [ https://issues.apache.org/jira/browse/HDFS-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374535#comment-15374535 ] 

Ravikumar commented on HDFS-3051:
---------------------------------

How about returning the MappedByteBuffers of all blocks for a file in local. If there are non-local blocks, this method can simply return empty.

public List<ByteBuffer> readFullyScatterGatherLocal(EnumSet<ReadOption> options)
    throws IOException {
    return ((PositionedReadable)in).readFullyScatterGather(options);
}

A quick sample-impl can be like

public List<ByteBuffer> readFullyScatterGatherLocal(EnumSet<ReadOption>) throws IOException
{
  List<LocatedBlock> blockRange = getBlockRange(0, getFileLength());
  if(!allBlocksInLocal(blockRange)) 
 {
    return;
 }
 List<ByteBuffer> retval = new LinkedList<ByteBuffer>();
 for(LocatedBlock blk:blockRange) 
 {
   blkReader = fetchBlockReader(blk, localDNAddrPair);
  ClientMmap mmap = blkReader.getClientMmap(readOptions);
  mmap.setunmap(false); //Instruction to cache-eviction to avoid unmapping this. Slots, streams & all other resources will be closed
  result.add(mmap.getMappedByteBuffer());
  closeBlockReader(blkReader);
}
return retval
}

Apps opening InputStreams only once (Hbase??) can call this method & use the zero-copy buffers for reads, if file is local.  If not available, they can fall back to regular DFSInputStream. Reads can eliminate sync overheads & get same perf as a local filesystem.

But I don't know if "leaking" MappedByteBuffers to calling code can have nasty side-effects. 






> A zero-copy ScatterGatherRead api from FSDataInputStream
> --------------------------------------------------------
>
>                 Key: HDFS-3051
>                 URL: https://issues.apache.org/jira/browse/HDFS-3051
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client, performance
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> It will be nice if we can get a new API from FSDtaInputStream that allows for zero-copy read for hdfs readers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org