From user-return-4243-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Wed Apr 01 19:35:18 2009 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 44252 invoked from network); 1 Apr 2009 19:35:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Apr 2009 19:35:15 -0000 Received: (qmail 38105 invoked by uid 500); 1 Apr 2009 19:35:14 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 38017 invoked by uid 500); 1 Apr 2009 19:35:14 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 38007 invoked by uid 99); 1 Apr 2009 19:35:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 19:35:14 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [68.142.200.126] (HELO web30603.mail.mud.yahoo.com) (68.142.200.126) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 01 Apr 2009 19:35:06 +0000 Received: (qmail 25974 invoked by uid 60001); 1 Apr 2009 19:34:44 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1238614484; bh=2Pn9rrK/SbaP9s8FeCVZmkLwTXok6/eBfIsFuhj7vbI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=6SOh7vSVpvSXzJAEms8oZZ6CQZnn2zl6zbMiO2epa//SBmAnKf5DLemkdkijGqy7f0v9bxrl5WUupxInN3Ao7DxR/3xVrV/6tlL7x55yS9BtA76Tu7H3NqTaPDCdDdMw5X900Kv/OPOclVVQBj/UmzNDSA/KFvf3kkrk9o1lHAM= DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=yTgaIpQ8x1L15I9Be95fvCW49fRlrzHT8cE80iTjz+F93TmX99EMOstiH/CUczNprurOTlq0rJ99rjM64lswXlAk0hxiNA8Qr2tYQhQCTpyaks6Kv/PFUZ8PSvkqV4iy4i3kWTkmxXQwCwjsTSulkUnUQMzXTpeaVx5UNYOLVLM=; Message-ID: <719071.24295.qm@web30603.mail.mud.yahoo.com> X-YMail-OSG: n2_6VEYVM1mSL13VCksod4qME5slBcybL1O.RPV68u8vU6vNNhNZ56IhGzKLQQ9_loifFsCP9laJTkhyC8RTCt7FQaxlXEdStCFV2Y6CYsafvn17T0s28jqbthE5qOcBUjYnjbpIYtobsjFGk_E7vAMt.DtfPWxeBv2aae86.SAj1W9MsyVzAwd8pwPsy_FHR9w4IH8Y0VvYhhp2wxbt9BiNbRQ90pvEzb5hEUGqx3I5UvXL.u4vx6V.VPlErHRFwcMNQ8eJoLyM Received: from [59.92.177.27] by web30603.mail.mud.yahoo.com via HTTP; Wed, 01 Apr 2009 12:34:44 PDT X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1 Date: Wed, 1 Apr 2009 12:34:44 -0700 (PDT) From: Manjunath Somashekhar Subject: Suggestions on optimizing document look up To: user@couchdb.apache.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Checked: Checked by ClamAV on apache.org hi All, Buoyed by the response i got to my previous mail (Suggestions on View performance optimization/improvement), i am asking another question for optimizing document look up based on _id. Let us say we have a db containing a million documents each with _id generated by us [1.....1000000]. If we have to get all the documents one by one (assuming the search/lookup code will get random inputs of [1..1000000]), wat would work best? As of now wat we are doing is a simple look up like: def getDocById(self, id): return self.db[id] For doing a million lookups like this it takes about 50-60 mins on my laptop. Is there a better way of doing the same? Thought of fetching a bunch of keys in one go caching them (LRU style) and looking up the cache first before hitting the db, but given that the input 'id' randomly varies between [1..1000000], it has not been a great success. Any thoughts? Ideas? Suggestions? Environment details: Couchdb - 0.9.0a757326 Erlang - 5.6.5 Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux Ubuntu distribution Centrino Dual core, 4GB RAM laptop Thanks Manju