Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Date: Sat, 26 Jan 2013 14:25:12 +0000 (UTC)
From: "Joan Touzet (JIRA)" <jira@apache.org>
To: dev@couchdb.apache.org
Message-ID: <JIRA.12415848.1235811808681.195924.1359210312639@arcas>
In-Reply-To: <JIRA.12415848.1235811808681@arcas>
References: <JIRA.12415848.1235811808681@arcas>
Subject: [jira] [Commented] (COUCHDB-271) preventing compaction from ruining
 the OS block cache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13563498#comment-13563498 ] 

Joan Touzet commented on COUCHDB-271:
-------------------------------------

Recommend pushing this farther out, this is non-trivial, rnewson's suggestion is massively non-backward-compatible.

This is not a good candidate for a (soon now!) 1.3.0.
                
> preventing compaction from ruining the OS block cache
> -----------------------------------------------------
>
>                 Key: COUCHDB-271
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-271
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>    Affects Versions: 0.8.1, 0.9
>            Reporter: Jan Lehnardt
>             Fix For: 1.3
>
>
> Adam Kocolosk:
> Hi, I've noticed that compacting large DBs pretty much kills any filesystem caching benefits for CouchDB.  I believe the problem is that the OS (Linux 2.6.21 kernel in my case) is caching blocks from the .compact file, even though those blocks won't be read again until compaction has finished.  In the meantime, the portion of the cache dedicated to the old DB file shrinks and performance really suffers.
> I think a better mode of operation would be to advise/instruct the OS not to cache any portion of the .compact file until we're ready to replace the main DB.  On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to go:
> http://linux.die.net/man/2/posix_fadvise
> This link has a little more detail and a usage example:
> http://insights.oetiker.ch/linux/fadvise.html
> Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang VM.  Perhaps the simplest approach would be to have a helper process that we can spawn which calls that function (or its equivalent on a non-Linux OS) periodically during compaction?  I'm not really sure, but I wanted to get this out on the list for discussion.  Best,

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira