couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: recovering data from an unfinished compaction db
Date Mon, 24 Sep 2012 16:45:46 GMT
The compactor is written to flush batches of docs every 5K bytes and
then write a header out ever 5M bytes (assuming default batch sizes).
Its important to remember that this judged against #doc_info{} records
which don't contain a full doc body. For documents with relatively few
revisions we're looking at (rough guess) ~100 bytes per record, which
is going to give us 50K docs per header commit. Seeing as the OP
mentions lots of attachments this could give us a relatively large gap
in the file to search for a header.

On Mon, Sep 24, 2012 at 11:17 AM, Tim Tisdall <tisdall@gmail.com> wrote:
> Since this is the result of a compaction, shouldn't the header be at
> the beginning of the file?  (just testing my knowledge on how all this
> works...)
>
> -Tim
>
> On Mon, Sep 24, 2012 at 12:09 PM, Robert Newson <rnewson@apache.org> wrote:
>> That does imply that the last valid header is a long way back up the
>> file, though.
>>
>> On 24 September 2012 17:00, Paul Davis <paul.joseph.davis@gmail.com> wrote:
>>> I'd ignore the snappy error for now. There's no way this thing ran for
>>> an hour and then suddenly hit an error in that code. If this is like a
>>> bug I've seen before the reason that this runs out of RAM is due to
>>> the code that's searching for a header not releasing binary ref counts
>>> as it should be.
>>>
>>> The quickest way to fix this would probably be to go back and update
>>> recover-couchdb to recognize the new disk format. Although that gets
>>> harder now that snappy compression is involved.
>>>
>>> On Mon, Sep 24, 2012 at 10:32 AM, Dave Cottlehuber <dch@jsonified.com>
wrote:
>>>> On 24 September 2012 15:02, Robert Newson <rnewson@apache.org> wrote:
>>>>> {badmatch,{error,snappy_nif_not_loaded} makes me wonder if this 1.2
>>>>> installation is even right.
>>>>>
>>>>> Can someone enlighten me? Is it possible to get this error spuriously?
>>>>
>>>> No. I'd be keen to see a bit of logfiles to understand what's not working.
>>>>
>>>>> Does running out of RAM cause erlang to unload NIF's?
>>>>
>>>> I don't think so on Windows.
>>>>
>>>> There's an R15B01 based build here:
>>>> https://www.dropbox.com/sh/jeifcxpbtpo78ak/GG9fjWOyDt/Snapshots/20120524
>>>> that has a fix for a more recent version of Windows server than I have
>>>> to address one NIF loading error, although there are a number of
>>>> possible causes.
>>>>
>>>> @Rudi can you give this a go & report back?
>>>>
>>>> A+
>>>> Dave

Mime
View raw message