hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5182) BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid
Date Tue, 04 Feb 2014 23:08:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891378#comment-13891378
] 

Haohui Mai commented on HDFS-5182:
----------------------------------

I should have said it more concretely. What I'm proposing is that the DN passes the file descriptor
to the client (e.g., using {{ioctl()}} in Linux).

It seems to me that with this approach (1) the OS takes cares about the resource management,
and (2) the client has more flexibility. The client can access the file using the {{read()}}
and {{write()}} system calls. The client, of course, can calls {{mmap()}} of the descriptor
to implement zero-copy reads with respect to its process boundary. It can also call {{ioctl()}}
and {{madvise()}} to specify the OS buffer cache policy of the file. The additional flexibility
can be quite useful to for implementing databases on HDFS.

> BlockReaderLocal must allow zero-copy  reads only when the DN believes it's valid
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-5182
>                 URL: https://issues.apache.org/jira/browse/HDFS-5182
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>
> BlockReaderLocal must allow zero-copy reads only when the DN believes it's valid.  This
implies adding a new field to the response to REQUEST_SHORT_CIRCUIT_FDS.  We also need some
kind of heartbeat from the client to the DN, so that the DN can inform the client when the
mapped region is no longer locked into memory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message