Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of stephenp@agrussell.com
 designates 204.251.33.236 as permitted sender)
Message-Id: <BAF96C1A-47DC-46F3-B073-BD71096FE97D@agrussell.com>
From: Stephen Prater <stephenp@agrussell.com>
To: user@couchdb.apache.org
In-Reply-To: <DDB4E9C6-7E27-44A9-9EF6-166C1C5CEDAD@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Seamless view rebuilding?
Date: Thu, 26 Aug 2010 14:55:45 -0500
References: <4C766ECA.1090100@gmail.com>
 <AANLkTiny+SROYzd+YdbGroN-6Dnnaj1YhYmATW+xAY6e@mail.gmail.com>
 <726DB993-42A3-41E4-9F4A-D7D10D8A8AC1@apache.org>
 <4C767D8F.7000704@gmail.com>
 <DDB4E9C6-7E27-44A9-9EF6-166C1C5CEDAD@apache.org>

I solved a somewhat similar problem (where I needed a new view index  
on every push, regardless of whether the view had actually changed) by  
adding a !stamp macro to couchapp which subs in a comment with the  
timestamp.

You could pull the new ddoc copy trick then - since it would  
effectively be a new view each time you did a couchapp push.

Of course, that rebuilds the ENTIRE view, which might not be okay.

On Aug 26, 2010, at 10:00 AM, J Chris Anderson wrote:

>
> On Aug 26, 2010, at 7:43 AM, Patrick Barnes wrote:
>
>> What about a staging db?
>>
>> If I had:
>> * Continuous replication from production->staging (for those  
>> piecemeal updates)
>> * Made major batch updates to the staging server
>> * Rebuilt the staging server views, then
>> * Replicated from staging->production
>>
>
> View files don't replicate, so this won't help for the batches.
>
> Best bet is probably to query the view periodically during the batch  
> import.
>
> Chris
>
>> Would the production db's views take as long to rebuild as the  
>> single-db model, or does it have some mechanism to optimise it?
>>
>> The couchdb server is also behind a proxy, so there might be some  
>> solution there?
>>
>> -PB
>>
>>
>> On 26/08/2010 11:48 PM, Adam Kocoloski wrote:
>>> But he doesn't have a new view, just a very large batch of updates  
>>> added to an existing view.
>>>
>>> On Aug 26, 2010, at 9:46 AM, Robert Newson wrote:
>>>
>>>> Create a new ddoc with your new view, query that view, waiting  
>>>> for it
>>>> to build, and then copy your new ddoc over your old one. View  
>>>> indexes
>>>> are named on disk after their digest specifically to allow this
>>>> offline building feature. :)
>>>>
>>>> B.
>>>>
>>>> On Thu, Aug 26, 2010 at 2:40 PM, Patrick  
>>>> Barnes<mrtrick@gmail.com>  wrote:
>>>>> I have a database serving documents through a number of  
>>>>> intermediary
>>>>> application servers, to the users' web browsers.
>>>>>
>>>>> There are two mechanisms by which documents are modified;
>>>>> a) Piecemeal updates as a result of user actions. (ie adding or  
>>>>> updating a
>>>>> record)
>>>>> b) Bulk updates, typically from import scripts, that might  
>>>>> modify tens of
>>>>> thousands of documents at once.
>>>>>
>>>>> The problem I'm having, is that when a set of bulk updates go  
>>>>> through, it
>>>>> can take a long time to rebuild the view indexes. Meanwhile,  
>>>>> several user's
>>>>> web requests will time out until rebuilding is complete.
>>>>>
>>>>> Stale=ok is a simple solution to the bulk problem, but the  
>>>>> application
>>>>> servers will also expect to be able to update documents, and  
>>>>> retrieve the
>>>>> changes immediately after.
>>>>>
>>>>> Is there a good way to avoid these large view update delays?
>>>>>
>>>>> -Patrick Barnes
>>>>>
>>>
>>>
>
>
>