From dev-return-19118-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Sun Nov 6 14:53:17 2011 Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 138A59413 for ; Sun, 6 Nov 2011 14:53:17 +0000 (UTC) Received: (qmail 25710 invoked by uid 500); 6 Nov 2011 14:53:15 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 25646 invoked by uid 500); 6 Nov 2011 14:53:15 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 25538 invoked by uid 99); 6 Nov 2011 14:53:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Nov 2011 14:53:15 +0000 X-ASF-Spam-Status: No, hits=-2001.2 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Nov 2011 14:53:12 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id BE96234E26 for ; Sun, 6 Nov 2011 14:52:51 +0000 (UTC) Date: Sun, 6 Nov 2011 14:52:51 +0000 (UTC) From: "Filipe Manana (Updated) (JIRA)" To: dev@couchdb.apache.org Message-ID: <738358239.4727.1320591171781.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1161314718.4725.1320591171670.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Updated] (COUCHDB-1334) Indexer speedup (for non-native view servers) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/COUCHDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Filipe Manana updated COUCHDB-1334: ----------------------------------- Attachment: master-0002-More-efficient-communication-with-the-view-server.patch 0002-More-efficient-communication-with-the-view-server.patch 0001-More-efficient-view-updater-writes.patch > Indexer speedup (for non-native view servers) > --------------------------------------------- > > Key: COUCHDB-1334 > URL: https://issues.apache.org/jira/browse/COUCHDB-1334 > Project: CouchDB > Issue Type: Improvement > Components: Database Core, JavaScript View Server, View Server Support > Reporter: Filipe Manana > Assignee: Filipe Manana > Fix For: 1.2 > > Attachments: 0001-More-efficient-view-updater-writes.patch, 0002-More-efficient-communication-with-the-view-server.patch, master-0002-More-efficient-communication-with-the-view-server.patch > > > The following 2 patches significantly improve view index generation/update time and reduce CPU consumption. > The first patch makes the view updater's batching more efficient, by ensuring each btree bulk insertion adds/removes a minimum of N (=100) key/value pairts. This also makes the index file size grow not so fast with old data (old btree nodes basically). This behaviour is already done in master/trunk in the new indexer (by Paul Davis). > The second patch maximizes the throughput with an external view server (such as couchjs). Basically it makes the pipe (erlang port) communication between the Erlang VM (couch_os_process basically) and the view server more efficient since the 2 sides spend less time block on reading from the pipe. > Here follow some benchmarks. > test database at http://fdmanana.iriscouch.com/test_db (1 million documents) > branch 1.2.x > $ echo 3 > /proc/sys/vm/drop_caches > $ time curl http://localhost:5984/test_db/_design/test/_view/test1 > {"rows":[ > {"key":null,"value":1000000} > ]} > real 2m45.097s > user 0m0.006s > sys 0m0.007s > view file size: 333Mb > CPU usage: > $ sar 1 60 > 22:27:20 %usr %nice %sys %idle > 22:27:21 38 0 12 50 > (....) > 22:28:21 39 0 13 49 > Average: 39 0 13 47 > branch 1.2.x + batch patch (first patch) > $ echo 3 > /proc/sys/vm/drop_caches > $ time curl http://localhost:5984/test_db/_design/test/_view/test1 > {"rows":[ > {"key":null,"value":1000000} > ]} > real 2m12.736s > user 0m0.006s > sys 0m0.005s > view file size 72Mb > branch 1.2.x + batch patch + os_process patch > $ echo 3 > /proc/sys/vm/drop_caches > $ time curl http://localhost:5984/test_db/_design/test/_view/test1 > {"rows":[ > {"key":null,"value":1000000} > ]} > real 1m9.330s > user 0m0.006s > sys 0m0.004s > view file size: 72Mb > CPU usage: > $ sar 1 60 > 22:22:55 %usr %nice %sys %idle > 22:23:53 22 0 6 72 > (....) > 22:23:55 22 0 6 72 > Average: 22 0 7 70 > master/trunk > $ echo 3 > /proc/sys/vm/drop_caches > $ time curl http://localhost:5984/test_db/_design/test/_view/test1 > {"rows":[ > {"key":null,"value":1000000} > ]} > real 1m57.296s > user 0m0.006s > sys 0m0.005s > master/trunk + os_process patch > $ echo 3 > /proc/sys/vm/drop_caches > $ time curl http://localhost:5984/test_db/_design/test/_view/test1 > {"rows":[ > {"key":null,"value":1000000} > ]} > real 0m53.768s > user 0m0.006s > sys 0m0.006s -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira