incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Stockton <chrisstockto...@gmail.com>
Subject Re: Bug or my lack of understanding? "Reduce output must shrink more rapidly"
Date Wed, 17 Aug 2011 17:24:11 GMT
Hello,

On Tue, Aug 16, 2011 at 11:58 PM, Marcello Nuccio
<marcello.nuccio@gmail.com> wrote:
> I don't think this is a bug. Citing from
> http://guide.couchdb.org/draft/cookbook.html#aggregate
>
>    As a rule of thumb, the reduce function should reduce to a single
> scalar value. That is, an integer; a string; or a small, fixed-size
> list or object that includes an aggregated value (or values) from the
> values argument.
>
> The object returned by your reduce function, is not fixed-size.
> Actually it is bigger than the input document.
>
> Marcello
>

I actually think we fit the rule of thumb in the sense the object
returned is fixed size object of scalars. It just so happens to have
lots of keys. The reason this is necessary is because we do much more
then just the total, we do the population variance, population
standard deviation, square, sums, the average total including nulls,
and the average total not including nulls.

I wont go too deep into our applications architecture but this record
represents a "row" in a list of user data with different kinds of data
types, typically users don't want any kinds of stats etc, so we just
serve out the documents. However some users want statistical
information about their data, so in those cases we run the reduce, it
is a win because we have 1 view for both the map and reduce. To
redesign our map to emit columns means a separate view for statistics,
which means a couple things that I am not okay with, like double the
disk space, multiple view calls per statistics load.

At the end of the day I think I disagree that this is not a bug, I
think there is room for improvement in this area and should be
discussed. If this is not the appropriate place to do so I can write a
jira ticket.

-Chris

Mime
View raw message