hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
Date Thu, 27 Oct 2011 22:36:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137667#comment-13137667

Todd Lipcon commented on HDFS-2465:

bq. There isn't any config that enables syncBehindWrites in BlockerRecevier
Woops! good catch. Lost that somewhere along the line.

bq. Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet
size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS
I don't think so. HDFS doesn't really do readahead except for filling up the TCP window. But
submitting very large (multi-MB)readahead to the OS lets it schedule big sequential reads.
So, I don't think there's any real correlation. To set this tunable, I'd probably do some
mental math like: "well, I probably have 20 files being read at once, so 8M of readahead on
each of those will end up being 160M of RAM used." Having to multiply by other factors is
harder to follow.

Fixed the other nits. New patch momentarily.
> Add HDFS support for fadvise readahead and drop-behind
> ------------------------------------------------------
>                 Key: HDFS-2465
>                 URL: https://issues.apache.org/jira/browse/HDFS-2465
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-2465.txt
> This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and
should be considered experimental, as discussed in the parent JIRA. It should be off by default
until better heuristics, APIs, and tuning experience is developed.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message