couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: View re-building from 0.9.1 to 0.10.0
Date Mon, 26 Oct 2009 16:19:20 GMT

On 26 Oct 2009, at 17:08, Chris Anderson wrote:

> On Mon, Oct 26, 2009 at 7:58 AM, till <till@php.net> wrote:
>> Hey guys,
>>
>> I've tested this a couple times with smaller database and so far, I
>> always had to rebuild my views when I updated a server from 0.9.x to
>> 0.10.0. And I also had that confirmed (by some of you). ;-)
>>
>> Now my problem is the following -- last time I build the view on the
>> database, it took a week to finish, literally 7.5 days. The view code
>> itself is pretty simple, but the database contains a 160,000,000
>> documents and totals to ~200 GB in size. With a set of this  
>> magnitude,
>> I'd like to avoid another week. I'd even like to avoid 3-4 days (even
>> though I'm a fan, I don't think it's twice as fast).
>>
>> So my question is if there is *any* way to avoid view rebuilding and
>> still updating to 0.10.x.
>>
>
> The principle difference between the 0.9.x code and the 0.10.x code is
> the view index filename. When you update to 0.10.0, and start couchdb
> building the view, it will create a file named after a hash of the
> ddoc.views contents. If you restart the new couch, and rename the 0.9
> view file to the hash name, and then start 0.10.0, there's a chance it
> could work.

> Obviously this should be tested, and perhaps Jan already tried that.
> There may be differences is the header format between versions that
> should accounted for. This part of the patch (if it is needed) will be
> more challenging.

I did a little patch for Till that looks for 0.9-filenames when it  
couldn't
find the 0.10 md5 file. That worked until 0.10 found out that the header
didn't match the 0.10 format and reset the index.

I stopped short of diving into the update_db_header() function.

Someone with a little time on his/her hands and some erlang-fu
can pull this off pretty easily. My code so far is on GitHub:

http://github.com/janl/couchdb/tree/zero-nine-indexes-for-oh-ten

I'd be in favour of adding this to 0.10.1.

Cheers
Jan
--


> All in all, I'm guessing this is doable for 0.10.1, but it's not
> guaranteed to be easy.
>
> We can' allow this kinda thing between beta releases, and 0.10.1 is a
> good chance to get the 0.9 update process solid.
>
> Hope that helps,
> Chris
>
> ps. Damien did the 4-5x map/reduce speedup on 0.11 (current trunk), so
> if you're gonna do a big rebuild, that's worth looking at.
>
> Devs - what we need is a callback that modules can use to do things
> like rename files on upgrade.
>
>> Thanks for listening,
>> Till
>>
>
>
>
> -- 
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>


Mime
View raw message