Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 3 Apr 2012 18:42:23 +0000 (UTC)
From: "Henry Robinson (Updated) (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1555178803.7674.1333478543527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <855118746.26047.1331925940549.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HDFS-3110) libhdfs implementation of direct read
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Henry Robinson updated HDFS-3110:
---------------------------------

    Attachment: HDFS-3110.2.patch

Patch addressing Todd's concerns.

I added a 'flags' field to hdfsFile that has a bit set if a direct read is supported. I detect that by trying to issue a 0-byte read when the file is created.  If an exception is thrown, the flag is cleared, otherwise it is set. Once the flag is set, all subsequent hdfsRead calls will be diverted to hdfsReadDirect. 

An alternative is to use reflection to grab the input stream inside FsDataInputStream and use reflection to look for ByteBufferReadable, but that feels a little fragile (and complex to do in C); plus if some FS implements read(ByteBuffer) only to stub it out with a UnsupportedOperationException or similar, reads would never work correctly. 
                
> libhdfs implementation of direct read API
> -----------------------------------------
>
>                 Key: HDFS-3110
>                 URL: https://issues.apache.org/jira/browse/HDFS-3110
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: libhdfs
>            Reporter: Henry Robinson
>            Assignee: Henry Robinson
>             Fix For: 0.24.0
>
>         Attachments: HDFS-3110.0.patch, HDFS-3110.1.patch, HDFS-3110.2.patch
>
>
> Once HDFS-2834 gets committed, we can add support for the new API to libhdfs, which leads to significant performance increases when reading local data from C.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira