hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cristina L. Abad (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7832) Using mincore to understand the effect of fadvise in the Linux page cache
Date Thu, 17 Nov 2011 21:26:54 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152342#comment-13152342
] 

Cristina L. Abad commented on HADOOP-7832:
------------------------------------------

At this point I just wanted to share it. It can be very useful i evaluating the fadvise effect
on the page cache. For example, I ran some simple tests and verified the benefit of using
fadvise: for the intermediate files of a simple terasort test (1 slave node, 10G of input
data), the fadvise stuff leads to an increase of from 44% to 78% of the average of pages (of
intermediate files) already in the page cache. This is an effect of Map input data being flushed
out of the page cache periodically. In this case, I am talking about the intermediate files
that we ideally want to keep cached as long as possible (before being sent to the reducers)
so that the shuffle and sort phase finishes faster. You already showed how this improves job
completion time, and this patch shows how/why this happens.
                
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7832
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io, native, performance
>            Reporter: Cristina L. Abad
>            Priority: Minor
>         Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux
page cache (when the file is being opened) with the purpose of measuring the effect of issuing
posix_fadvise (POSIX_FADV_DONTNEED) calls.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message