Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of skippy.hammond@gmail.com
 designates 209.85.222.107 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=bzx86H6bPyq98VCeN81prBceTfHMwd6EGVxUfanAbxznrI4jVdp+6EIGvAi0btkfBe
         m4Eeb7SCAr0am+BvxgWSciNg0DOWL/yrjY6/rxjCtsSNQq3EczbuvF269dZYGqiNCkQf
         8m5wNjkXLZf7kc8ecg9/EMXm1S7iNfSisEHc0=
Message-ID: <4A0CC46D.3030009@gmail.com>
Date: Fri, 15 May 2009 11:25:01 +1000
From: Mark Hammond <skippy.hammond@gmail.com>
Reply-To: mhammond@skippinet.com.au
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US;
 rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2
MIME-Version: 1.0
To: dev@couchdb.apache.org
CC: Zachary Zolton <zachary.zolton@gmail.com>, B.Candler@pobox.com
Subject: Re: View Filter
References: <f728979f0905130741k7192039al30662aff8b2e45e4@mail.gmail.com>
 <20090514074329.GB6258@uk.tiscali.com>
 <f728979f0905140753vc8c721al5ecac3134be2acf1@mail.gmail.com>
 <20090514184701.GA18955@uk.tiscali.com>
In-Reply-To: <20090514184701.GA18955@uk.tiscali.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 15/05/2009 4:47 AM, Brian Candler wrote:
> On Thu, May 14, 2009 at 09:53:14AM -0500, Zachary Zolton wrote:
>> (1) people who are storing large documents in CouchDB but not indexing them
>> at all (I guess this is possible, e.g. if the doc ids are well-known or
>> stored in other documents, but this isn't the most common way of working)
>
> The proposal would exclude a document from *all* views in a particular
> design doc. So you're only going to get a benefit from this if you have a
> large number of documents (or a number of large documents) which are not
> required to be indexed in any view in that design doc.

Yep - and that is the point.  Consider Jan's example, where it was 
filtering on doc['type'].  If a database had (say) 10 potential values 
of 'type', then all filters that only care about a single type will only 
care about 1 in 10 of those documents.

Taking this to its extreme, we tested Jan's patch on a view which 
matches very few document in a large database.  Rebuilding that view 
with a filter was 18 times faster than without the filter.  We put this 
down to the fact the filter managed to avoid the json encode/decode step 
for the vast majority of the docs in the database.  IOW, on my test 
database, 6 minutes is spent before the filters can actually do anything 
(ie, that is just the json processing), whereas using the filter to 
avoid that json step brings it down to 20 seconds.

So while not everyone will be able to see such significant speedups, 
many may find it extremely useful.

> And it's reasonable, given that (as I understand it) each document is
> already only passed once to the view server, in order to be indexed by all
> the views in that design document.

I agree there is lots that can and should be done to speed up views that 
do indeed care about most of the docs - such views spend less time 
relatively in the json encode step and more time in the interpreter.  As 
an experiment, I "ported" one of our views that does look at most of the 
docs from javascript to erlangview, and the performance increase was far 
more modest (20% maybe).  I suspect the javascript interpreter is faster 
than erlang, so I suspect that there will be a level of view complexity 
where using javascript *increases* view performance over erlang, even 
when factoring in the json processing...

Cheers,

Mark