hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Robinson (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3110) libhdfs implementation of direct read API
Date Tue, 03 Apr 2012 18:42:23 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Henry Robinson updated HDFS-3110:
---------------------------------

    Attachment: HDFS-3110.2.patch

Patch addressing Todd's concerns.

I added a 'flags' field to hdfsFile that has a bit set if a direct read is supported. I detect
that by trying to issue a 0-byte read when the file is created.  If an exception is thrown,
the flag is cleared, otherwise it is set. Once the flag is set, all subsequent hdfsRead calls
will be diverted to hdfsReadDirect. 

An alternative is to use reflection to grab the input stream inside FsDataInputStream and
use reflection to look for ByteBufferReadable, but that feels a little fragile (and complex
to do in C); plus if some FS implements read(ByteBuffer) only to stub it out with a UnsupportedOperationException
or similar, reads would never work correctly. 
                
> libhdfs implementation of direct read API
> -----------------------------------------
>
>                 Key: HDFS-3110
>                 URL: https://issues.apache.org/jira/browse/HDFS-3110
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: libhdfs
>            Reporter: Henry Robinson
>            Assignee: Henry Robinson
>             Fix For: 0.24.0
>
>         Attachments: HDFS-3110.0.patch, HDFS-3110.1.patch, HDFS-3110.2.patch
>
>
> Once HDFS-2834 gets committed, we can add support for the new API to libhdfs, which leads
to significant performance increases when reading local data from C.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message