From user-return-13416-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Sat Oct 30 13:53:20 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 48091 invoked from network); 30 Oct 2010 13:53:20 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 30 Oct 2010 13:53:20 -0000 Received: (qmail 12134 invoked by uid 500); 30 Oct 2010 13:53:18 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 11953 invoked by uid 500); 30 Oct 2010 13:53:18 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 11945 invoked by uid 99); 30 Oct 2010 13:53:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 13:53:17 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of anandology@gmail.com designates 209.85.161.52 as permitted sender) Received: from [209.85.161.52] (HELO mail-fx0-f52.google.com) (209.85.161.52) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 13:53:11 +0000 Received: by fxm12 with SMTP id 12so3804988fxm.11 for ; Sat, 30 Oct 2010 06:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:from:date :message-id:subject:to:content-type; bh=5FgrG/pqhULqq/ZGVqRBEkQcPi1GHS7LmD613MoIf50=; b=hgaWReqIjUQZ+U8dCa5qMuftSoHVtyAD2Zrbnt7duAHO7HXPaneeweW5vbdJlOS0CO TPRANroXomVnWJ5tLIl+ObjKLFGOVAcPP1JgzL8b7ce5aHO3JSsTGgNUdNfqhgcbUOK5 dLSqwaPgEAr13RzKEDQiaL8b3seeBZ1KAEh3Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; b=hdQpGpHbHGkTrhCw4NE82C2j4fdBir8S44iq/TBTYcg404E9v5foPMyLZ5go0yeMTZ RntA8kWB9yOnaJqS3O2/YC4WlpAi/qxhonssPyisrasaca0kwD/kRRubaDfNpSbmzs5M kMjo9qxoSirwETP295k04wm4RH61ZToPscjSs= Received: by 10.223.83.13 with SMTP id d13mr269456fal.56.1288446770206; Sat, 30 Oct 2010 06:52:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.147.82 with HTTP; Sat, 30 Oct 2010 06:52:30 -0700 (PDT) From: Anand Chitipothu Date: Sat, 30 Oct 2010 19:22:30 +0530 Message-ID: Subject: How to speedup view generation? To: user Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Hi, I'm trying to setup a couchdb database with 14M documents. The view generation is taking too long. It is running at the rate of 22 docs/sec right now. At this rate it will take 7days to build the view, which is too slow and I expect the speed to go down further as the view file size increase. Is there any way to speedup the view generation? Is it possible to parallelize it? Here is what I've done till now: * I tried writing the view in javascript, python and erlang and the python version seems to be working faster, so I'm using that. * I've presorted the documents by key before loading, so the view updater must be reading the docs from disk sequentially. * I've put the db file and view files on separate disks. You can look at my view and sample docs here: http://github.com/anandology/ol-couch/tree/master/python/views/lists http://github.com/anandology/ol-couch/blob/master/sample_docs.txt I'm working with couchdb 1.0.1 on Linux server. I'm struggling to improve this from past 2 weeks with no luck. Do you have any tips to help me? Anand