hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7832) Using mincore to understand the effect of fadvise in the Linux page cache
Date Wed, 23 Nov 2011 16:07:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155935#comment-13155935
] 

Nathan Roberts commented on HADOOP-7832:
----------------------------------------

bq. On clusters with lots of blocks, we see block reports taking many minutes to generate
(and incurring lots of seeks) because the FS metadata gets evicted when MR jobs run.

Todd, have you experimented with /proc/sys/vm/vfs_cache_pressure to deal with this? Seems
like datanodes might want to have this value set very low so that dentry/inode caches are
significantly stickier than page cache pages. I'm having a hard time coming up with scenarios
where this would be detrimental. 

                
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7832
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io, native, performance
>            Reporter: Cristina L. Abad
>            Priority: Minor
>         Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux
page cache (when the file is being opened) with the purpose of measuring the effect of issuing
posix_fadvise (POSIX_FADV_DONTNEED) calls.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message