hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4953) enable HDFS local reads via mmap
Date Mon, 15 Jul 2013 20:10:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708905#comment-13708905
] 

Uma Maheswara Rao G commented on HDFS-4953:
-------------------------------------------

 {code}
 if (close) {
-        out.close();
         out = null;
+        out.close();
         in.close();
{code}
I think you have changed this by mistake?

I think, if we read different blocks data here, we need to map for each separately right?
So, cache gets filled quickly and what is the result with this kind of scenario?

Also for smaller files, this may be overhead right as mapping overhead may be more than reads.
(can add minimum block size for mmap?)

Did you try implementing native call for unmap instead of depending on Sun package. In fact
I too introduced dependency on this cleaner code when I try introducing this on write path
at DNs in my internal branch. :-).



                
> enable HDFS local reads via mmap
> --------------------------------
>
>                 Key: HDFS-4953
>                 URL: https://issues.apache.org/jira/browse/HDFS-4953
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>    Affects Versions: 2.2.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: benchmark.png, HDFS-4953.001.patch, HDFS-4953.002.patch
>
>
> Currently, the short-circuit local read pathway allows HDFS clients to access files directly
without going through the DataNode.  However, all of these reads involve a copy at the operating
system level, since they rely on the read() / pread() / etc family of kernel interfaces.
> We would like to enable HDFS to read local files via mmap.  This would enable truly zero-copy
reads.
> In the initial implementation, zero-copy reads will only be performed when checksums
were disabled.  Later, we can use the DataNode's cache awareness to only perform zero-copy
reads when we know that checksum has already been verified.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message