From dev-return-16271-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Mon May 23 17:37:28 2011 Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ED057462E for ; Mon, 23 May 2011 17:37:28 +0000 (UTC) Received: (qmail 43948 invoked by uid 500); 23 May 2011 17:37:28 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 43914 invoked by uid 500); 23 May 2011 17:37:28 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 43906 invoked by uid 99); 23 May 2011 17:37:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:37:28 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:37:27 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 59D57D9D49 for ; Mon, 23 May 2011 17:36:47 +0000 (UTC) Date: Mon, 23 May 2011 17:36:47 +0000 (UTC) From: "Randall Leeds (JIRA)" To: dev@couchdb.apache.org Message-ID: <2087897852.36686.1306172207364.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1083053422.129.1298661502038.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Commented] (COUCHDB-1076) _all_docs performance degrades as doc_del_count increases MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/COUCHDB-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038083#comment-13038083 ] Randall Leeds commented on COUCHDB-1076: ---------------------------------------- Thanks, Filipe and Damien. Filipe: 1) I'd rather keep it in couch_db_updater. The other functions depending on (or generating) the reduction values for the by_id tree are in that module. It makes most sense to me to keep all the code that relies on that format in one place. 2) Cool. Easy fix. Good catch. 3) I'll see if I can add some trivial tests. > _all_docs performance degrades as doc_del_count increases > --------------------------------------------------------- > > Key: COUCHDB-1076 > URL: https://issues.apache.org/jira/browse/COUCHDB-1076 > Project: CouchDB > Issue Type: Bug > Reporter: Nathan Vander Wilt > > The time required to query _all_docs?limit=1 can be proportional to the doc_del_count of the database depending on where the deleted docs are in the sort order. As kocolosk explained on IRC: > The tree that serves as the source for _all_docs contains a small record for each document, deleted or not ... if you have a large number of deleted docs and their IDs are interspersed with the non-deleted ones i can imagine that it would cause additional seeks when streaming the _all_docs response > In my use case (https://github.com/natevw/RQMS) and in other cases (e.g. a rolling log or any long-lived database), the deleted docs may all be at the beginning of the _all_docs view, making query performance end up like using "?limit=N&skip=doc_del_count". > To improve the performance in cases where large blocks of documents have been deleted, kocolosk notes: > [10:30am] kocolosk: the inner nodes in the btree currently report the doc_count and doc_del_count > [10:31am] kocolosk: we might be able to rewrite the function that walks the btree so that it checks if the doc_count underneath an inner node is zero > [10:31am] kocolosk: and then it can skip that part of the tree entirely > [10:31am] kocolosk: instead of descending all the way to the leaf nodes and skipping deleted documents one by one > [10:31am] n[ate]vw: yeah, that'd have the benefit of being a code-only change > [10:31am] kocolosk: right > [10:32am] kocolosk: i think it should work rather nicely, though it probably requires some interesting spelunking into the depths of couch_btree > [10:33am] kocolosk: davisp: whaddya think? would what i'm proposing be possible? (checking for doc_count=0 in the inner node reduction and then skipping ahead when serving _all_docs) > ... > [10:39am] davisp: n[ate]vw: Its definitely doable, its just a question of how to do it so that I don't bleed from the eyes when trying to read through the implementation > ... > [10:40am] rnewson: davisp: right. we definitely want to preserve the high readability of the current btree code. /s > [10:40am] davisp: kocolosk: The only thing I can think of is passing a function to the view iteration that gets called to evaluate whether it should decend to a child node based on the key/reduction pair > [10:41am] kocolosk: davisp: what you're proposing would also allow for (more) efficient implementation of skip -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira