couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Barnes <mrtr...@gmail.com>
Subject Re: Reduces out of order?
Date Sun, 19 Jun 2011 07:44:09 GMT
It might be possible to work around this, depending on how typically 
out-of-order the documents are;

Have the reduce object be initially:
{
    result: "",
    version: 0,
    unapplied_deltas: [] //(a sorted list)
}
where each element of unapplied_deltas is:
{
    from:M,
    to:N,
    delta:"..."
}
and the delta documents contain a delta definition, the version number 
they expect to apply to, and the resulting version (usually from+1).

So the reduce function would look something like:

//Put the delta in the list, maintaining the sort order
obj.unapplied_deltas = insert_delta( obj.unapplied_deltas, doc );

//Can any deltas be applied?
while(
  obj.unapplied_deltas.length > 0 &&
  obj.version === obj.unapplied_deltas[0].from
) {
    d = obj.unapplied_deltas.shift();
    obj.result = apply_delta(obj.result, d);
    obj.version = d.to;
}

What do you think?

-Patrick




On 19/06/2011 7:45 AM, Charles Lehner wrote:
> Thanks for addressing my question, Dave and Sean. It looks like I can't do what I was
trying to do, which is to store deltas as documents and then use a time-ordered view to apply
the deltas and see the current state of the system, or its state at a point in the past.
>
> I could do this with a list function, but that would require iterating through all the
documents for each request. More likely I'll make something custom in node.js.
>
> Charles
>
> On Jun 17, 2011, at 10:18 PM, Dave Cottlehuber<dave@muse.net.nz>  wrote:
>
>> On 18 June 2011 14:07, Sean Copenhaver<sean.copenhaver@gmail.com>  wrote:
>>> Hmmm... I don't think it's a good idea to have an order sensitive reduce function.
Also I think you are thinking about the actual view index which is ordered by the key you
emit from the map function.
>>>
>>> I believe couch will reduce each b-tree node worth of the view and store those,
then rereduce those and parts of nodes as needed so your reduced queries only do part of the
work.
>>>
>>> I hope that made some sense.
>>>
>>>
>>>
>>> On Jun 17, 2011, at 9:17 PM, Charles Lehner<celehner1@gmail.com>  wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a reduce function that is sensitive to row order. My understanding
is that reduces are supposed to get their rows sorted by key. But this one is getting them
out of order; in fact, the order changes inconsistently between replications.
>>>>
>>>> I'm seeing this on both 1.0.2 and trunk. Is this the expected behavior?
>>>>
>>>> Thanks,
>>>> Charles
>>>
>>
>> Sean's spot on; MR should not be sensitive to row order; that's kinda
>> the point of being able to divide and conquer arbitrarily is that the
>> result is identical and independent of the order of processing.
>>
>> http://wiki.apache.org/couchdb/Introduction_to_CouchDB_views#Restrictions_on_map_and_reduce_functions
>>
>> I'm not sure what you meant by between replications, but after more
>> data is added to the couch, impacted b-tree leaf and intermediate
>> nodes get re-written. As its append-only, this continues right to the
>> top of the tree. So your "row order" changes each time more data is
>> added, potentially.
>>
>> Perhaps a list might be more appropriate for your post-processing
>> sorting? or client side?
>>
>> A+
>> Dave
>
>

Mime
View raw message