Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.212.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGxYWpbPPyEjQhTCYtjp2UJ=iaWVcaGuwcDeqw4Mth+QiVAWyA@mail.gmail.com>
References: 
 <CAFov+BLaetda+GhCK_T4JOwAMONCWcRnDQPknAfjs7MiL044Dg@mail.gmail.com>
 <CABvT1DF09qBkVRBVtrFfUo=MM1cRQB8wuQdUDkbphcg5asUYqA@mail.gmail.com>
 <CAFov+BJuUiaa2-9tpbuehm_ZeXkk2RHsbMHvVEVFj6_AZMPpTw@mail.gmail.com>
 <CABvT1DGa5bWGbafrsWQkWCospf-afLfhprCSG92S2+pde181tg@mail.gmail.com>
 <1341622793.20120924143243@whiletrue.com>
 <CABvT1DFEcqcUdAhYjuTJ_4FvNsij81ROtgF3HDJ-hj4_85g6AA@mail.gmail.com>
 <CAL+Y1ns=84vE5FeqUAoNPpjXnTeL_kjAmdHEHDjDpCW6p_kNUw@mail.gmail.com>
 <CAJ_m3YAEnjK5TpjhbYL0Qk5SzA8h42JPTnkZ3OZiiKQUQMv4VQ@mail.gmail.com>
 <CABvT1DGtNKXsqQ7MOS4aYXP_JqORRSk2oFP2eUfehcT77N4oHA@mail.gmail.com>
 <CAGxYWpbPPyEjQhTCYtjp2UJ=iaWVcaGuwcDeqw4Mth+QiVAWyA@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Mon, 24 Sep 2012 11:45:46 -0500
Message-ID: 
 <CAJ_m3YDh7eV+iTFp-Y=EDyH_q7w=dxvMwrJM-fgvBD5AG7gL8w@mail.gmail.com>
Subject: Re: recovering data from an unfinished compaction db
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The compactor is written to flush batches of docs every 5K bytes and
then write a header out ever 5M bytes (assuming default batch sizes).
Its important to remember that this judged against #doc_info{} records
which don't contain a full doc body. For documents with relatively few
revisions we're looking at (rough guess) ~100 bytes per record, which
is going to give us 50K docs per header commit. Seeing as the OP
mentions lots of attachments this could give us a relatively large gap
in the file to search for a header.

On Mon, Sep 24, 2012 at 11:17 AM, Tim Tisdall <tisdall@gmail.com> wrote:
> Since this is the result of a compaction, shouldn't the header be at
> the beginning of the file?  (just testing my knowledge on how all this
> works...)
>
> -Tim
>
> On Mon, Sep 24, 2012 at 12:09 PM, Robert Newson <rnewson@apache.org> wrote:
>> That does imply that the last valid header is a long way back up the
>> file, though.
>>
>> On 24 September 2012 17:00, Paul Davis <paul.joseph.davis@gmail.com> wrote:
>>> I'd ignore the snappy error for now. There's no way this thing ran for
>>> an hour and then suddenly hit an error in that code. If this is like a
>>> bug I've seen before the reason that this runs out of RAM is due to
>>> the code that's searching for a header not releasing binary ref counts
>>> as it should be.
>>>
>>> The quickest way to fix this would probably be to go back and update
>>> recover-couchdb to recognize the new disk format. Although that gets
>>> harder now that snappy compression is involved.
>>>
>>> On Mon, Sep 24, 2012 at 10:32 AM, Dave Cottlehuber <dch@jsonified.com> wrote:
>>>> On 24 September 2012 15:02, Robert Newson <rnewson@apache.org> wrote:
>>>>> {badmatch,{error,snappy_nif_not_loaded} makes me wonder if this 1.2
>>>>> installation is even right.
>>>>>
>>>>> Can someone enlighten me? Is it possible to get this error spuriously?
>>>>
>>>> No. I'd be keen to see a bit of logfiles to understand what's not working.
>>>>
>>>>> Does running out of RAM cause erlang to unload NIF's?
>>>>
>>>> I don't think so on Windows.
>>>>
>>>> There's an R15B01 based build here:
>>>> https://www.dropbox.com/sh/jeifcxpbtpo78ak/GG9fjWOyDt/Snapshots/20120524
>>>> that has a fix for a more recent version of Windows server than I have
>>>> to address one NIF loading error, although there are a number of
>>>> possible causes.
>>>>
>>>> @Rudi can you give this a go & report back?
>>>>
>>>> A+
>>>> Dave