Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of jchris@gmail.com designates
 209.85.220.226 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=FP7rR7HuhHmeD76troIOgujxyfKLTMJqP//4KMSxWsA7tvUvDh0tkj/1L9V7DWpqde
         Ok/k0Nj0QbqStFBa3gwzgcTOvnGl/6k4416UzIcW9sqV+Xjyj0aaY06rPngq1RAAzZk5
         kxEdOPgxFFTrji8lmmBY2znzxma/w6fBcevHs=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: Hitting the reduce overflow boundary
From: J Chris Anderson <jchris@gmail.com>
In-Reply-To: <ea2499da1003050823u2bbb9298t8ca944347a6ce1f2@mail.gmail.com>
Date: Fri, 5 Mar 2010 08:58:02 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <B2DBC1C2-B1B3-4400-9665-2A7CCA4F76A7@gmail.com>
References: <ea2499da1003020830g672d7e6dg6e913b83a22ab36d@mail.gmail.com>
  <71DE26DB-136C-43CD-B6C3-F51B00FC6C8F@googlemail.com>
 <ea2499da1003050317m68ad1beqd327b057ab9ffc02@mail.gmail.com>
 <ea2499da1003050823u2bbb9298t8ca944347a6ce1f2@mail.gmail.com>
To: dev@couchdb.apache.org


On Mar 5, 2010, at 8:23 AM, Dirkjan Ochtman wrote:

> On Fri, Mar 5, 2010 at 12:17, Dirkjan Ochtman <djc.ochtman@gmail.com> =
wrote:
>> I would really like to have someone from the dev team speak up on =
this
>> one, since I'd kind of like to re-enable the reduce_limit option, but
>> I don't think this view should be classified as overflowing.
>=20
> I happily found Adam in IRC, who explained this to me:
>=20
> 17:05 <+kocolosk> djc: so the current reduce_limit calculation is in =
main.js
> 17:06 <+kocolosk> the JSONified reduction needs to be less than 200 =
bytes, and
>                  it needs to be less than half of the size of the =
input map
>                  values
> 17:06 <+kocolosk> you could try tweaking those to see which condition =
you're
>                  failing
>=20
> The way I see it, the way the reduce phase should work is that the
> result from an collection of documents should be smaller or not much
> larger than the largest single object in the input set. This way,
> you'll prevent the unbounded growth that you want to prevent. Such a
> rule should also work on slightly larger inputs, because that should
> just be a larger constant, not exponentional growth.
>=20
> So I see two problems with the current rule:
>=20
> - it has a fixed limit at 200b, which isn't very reasonable because a
> larger size doesn't mean there's unbounded growth going on
> - it assumes that all the values in the input map have about equal
> size, which isn't really a requirement
>=20
> Am I crazy, or would a scheme like I proposed above be an improvement?

definitely. A patch to make the reduce_overflow_threshold configurable =
(with a default of 200 bytes) would be a major improvement and not hard =
to do.

Chris

>=20
> Cheers,
>=20
> Dirkjan