Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of kenneth.kalmer@gmail.com
 designates 209.85.218.170 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PigYdOo32p/IpyID7h+1vqwsIs2AsKVWzXdWPuM3WrJ9COQHWHTHlX4El/lXEPUu3x
         EQCHBAUwb4mq452L35BMPxUDUa/K/FuR2eJKofQ8FBCYzrDyYaYSaBEdaqzY/AzfGex6
         CrMUBPrPuqMx84n1oZXV9kDZCn1dQUHi63Z/E=
MIME-Version: 1.0
In-Reply-To: <e2111bbb0904090817j7ba10b34pe53ce2fd3c5c1590@mail.gmail.com>
References: <fad9d4840904090725h3852d1bcm61750dc1f4ca8d83@mail.gmail.com>
	 <e2111bbb0904090817j7ba10b34pe53ce2fd3c5c1590@mail.gmail.com>
Date: Sat, 11 Apr 2009 20:58:11 +0200
Message-ID: <fad9d4840904111158w4526a84fod43bab0d55f84450@mail.gmail.com>
Subject: Re: Some guidance with extremely slow indexing
From: Kenneth Kalmer <kenneth.kalmer@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=0016e6da98f209511f04674c100d

--0016e6da98f209511f04674c100d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Thu, Apr 9, 2009 at 5:17 PM, Paul Davis <paul.joseph.davis@gmail.com>wrote:

> Kenneth,
>
> I'm pretty sure you're issue is in the reduce steps for the daily and
> montly views. The general rule of thumb is that you shouldn't be
> returning data that grows faster than log(#keys processed) where as I
> believe your data is growing linearly with input.
>
> This particular limitation is a result of the implementation of
> incremental reductions. Basically, each key/pointer pair stores the
> re-reduced value for all [re-]reduce values in its children nodes. So
> as your reduction moves up the tree the data starts exploding which
> kills btree performance not to mention the extra file I/O.
>
> The basic moral of the story is that if you want reduce views like
> this per user you should emit a [user_id, date] pair as the key and
> then call your reduce views with group=true.
>
> HTH,
> Paul Davis
>

Hi Paul

Thanks for taking the trouble of investigating for me, I'll dive into the
views and clean them up a bit according to your advice as well as brush up
on the caveat you explained. I saw other threads in the archives where you
gave similar advice, sorry for not realizing I stepped into the same trap.
When I've got the issue resolved I'll update the gist and we can leave it as
a point of reference for others.

Thanks again!

-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

--0016e6da98f209511f04674c100d--