couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <>
Subject Re: View re-building from 0.9.1 to 0.10.0
Date Mon, 26 Oct 2009 16:19:20 GMT

On 26 Oct 2009, at 17:08, Chris Anderson wrote:

> On Mon, Oct 26, 2009 at 7:58 AM, till <> wrote:
>> Hey guys,
>> I've tested this a couple times with smaller database and so far, I
>> always had to rebuild my views when I updated a server from 0.9.x to
>> 0.10.0. And I also had that confirmed (by some of you). ;-)
>> Now my problem is the following -- last time I build the view on the
>> database, it took a week to finish, literally 7.5 days. The view code
>> itself is pretty simple, but the database contains a 160,000,000
>> documents and totals to ~200 GB in size. With a set of this  
>> magnitude,
>> I'd like to avoid another week. I'd even like to avoid 3-4 days (even
>> though I'm a fan, I don't think it's twice as fast).
>> So my question is if there is *any* way to avoid view rebuilding and
>> still updating to 0.10.x.
> The principle difference between the 0.9.x code and the 0.10.x code is
> the view index filename. When you update to 0.10.0, and start couchdb
> building the view, it will create a file named after a hash of the
> ddoc.views contents. If you restart the new couch, and rename the 0.9
> view file to the hash name, and then start 0.10.0, there's a chance it
> could work.

> Obviously this should be tested, and perhaps Jan already tried that.
> There may be differences is the header format between versions that
> should accounted for. This part of the patch (if it is needed) will be
> more challenging.

I did a little patch for Till that looks for 0.9-filenames when it  
find the 0.10 md5 file. That worked until 0.10 found out that the header
didn't match the 0.10 format and reset the index.

I stopped short of diving into the update_db_header() function.

Someone with a little time on his/her hands and some erlang-fu
can pull this off pretty easily. My code so far is on GitHub:

I'd be in favour of adding this to 0.10.1.


> All in all, I'm guessing this is doable for 0.10.1, but it's not
> guaranteed to be easy.
> We can' allow this kinda thing between beta releases, and 0.10.1 is a
> good chance to get the 0.9 update process solid.
> Hope that helps,
> Chris
> ps. Damien did the 4-5x map/reduce speedup on 0.11 (current trunk), so
> if you're gonna do a big rebuild, that's worth looking at.
> Devs - what we need is a callback that modules can use to do things
> like rename files on upgrade.
>> Thanks for listening,
>> Till
> -- 
> Chris Anderson

View raw message