Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.210.204 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=NoSCGTKi0DdfkSJL1tiQ3RT3m5Hojt+YZgXXHVfSD4btWXknQs+X/4GLGrS25CjVHF
         xx3ybnv/u7MPNWytN1cjlwiWtitHoHrQwzpvPTNJBsmB2+kEXRBWbk+S/uxYWp+X3WtC
         T6tEKtRBsOWz5LOgi68Gb7miCC3IH1lkg09ss=
MIME-Version: 1.0
In-Reply-To: <4A4FF0E4.4040804@krampe.se>
References: <4A4E7F53.7010406@krampe.se>
	 <e282921e0907031537h1628ee2fj4516d7540167e8ee@mail.gmail.com>
	 <F5B7CAE3-77CC-4046-902E-020D38B4DDA8@apache.org>
	 <4A4FF0E4.4040804@krampe.se>
Date: Sat, 4 Jul 2009 20:30:55 -0400
Message-ID: <e2111bbb0907041730m7c2a9b7h3e95f08fd1c540da@mail.gmail.com>
Subject: Re: Possible bug in indexer... (really)
From: Paul Davis <paul.joseph.davis@gmail.com>
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

2009/7/4 G=F6ran Krampe <goran@krampe.se>:
> Adam Kocoloski wrote:
>>
>> Not sure if it's described, but it is by design. =A0The reduce function
>> executes when the btree is modified. =A0We can't afford to cache KVs fro=
m an
>> index update in memory regardless of size; we have to set some threshold
>> when we flush them to disk.
>
> And I presume you can't write KVs *without* doing the reduce?
>
> When I wrote "described" I am referring to the blog post by Ricky Ho btw.=
 It
> seems to imply a strict ordering, map -> reduce -> rereduce. IIRC.
>

That was probably just the theoretical aspect. Map's always happen
first obviously, and then when the key/values are inserted into the
btree during a flush the entire tree is built which means that > 0
reduces are called and then re-reduces are run to fill out the tree.
At the moment we aren't delaying re-reduce calls because it'd require
a major overhaul to the btree code.

>> I think the fundamental question is why the flush operations were
>> occurring so frequently the second time around. =A0Is it because you wer=
e
>> building up a largish hash for the reduce value? =A0Probably. =A0Neverth=
eless,
>> I'd like to have a better handle on that.
>
> Yeah, well, I am on vacation now - but some other guys are not. We could =
of
> course start by trying to rewrite this the Right Way first as Chris said.
>
> I am curious if it can be done using grouping because we dismissed groupi=
ng
> due to its relatively slow performance (it runs lots of reduces at query
> time IIRC) :)
>
> Btw, the solution used now DOES return the map for a full year in about 2=
30
> ms, including parsing on client side. So query time was perfectly fine, b=
ut
> view generation was not. This shows to me that it *can* work.
>
> regards, G=F6ran
>
>