accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-140) Prevent linux from caching write ahead log files
Date Wed, 09 May 2012 15:07:50 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271475#comment-13271475
] 

Keith Turner commented on ACCUMULO-140:
---------------------------------------

I was looking into this a bit more yesterday.  Its shocking how difficult it is to tell the
linux kernel to not cache data.

http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html
https://issues.apache.org/jira/browse/CASSANDRA-1470

There are options to fadvise that linux simply ignores.  It honors fadvise DONTNEED, but this
only causes data already written to a file to be dropped from cache.  Looking at the HDFS
changes it seems that they periodically call fadvise dontneed as the file is written to. 
Besides using fadvise DONTNEED, direct IO seems to be another option.
                
> Prevent linux from caching write ahead log files
> ------------------------------------------------
>
>                 Key: ACCUMULO-140
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-140
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: logger
>            Reporter: Keith Turner
>            Assignee: Eric Newton
>             Fix For: 1.5.0
>
>
> While at ApacheCon I attended a talk about Cassandra.  The developer mentioned using
posix_fadvise to prevent linux from storing edit/write ahead log data in the page cache. 
This sounds great.  Accumulo can do this for its write ahead logs.
> Using posix_fadvise was also mentioned for compactions, this is something Accumulo can
not do because its compactions read and write to hdfs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message