Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAJ_m3YDBY1OEWRrXZiCpVDX1eF=DVMw4rCojAv-Z59Fs5vPKRg@mail.gmail.com>
References: 
 <CANpCgCPp_BetZL1yeAbSkpo_L2yzobRADn+_N77Oybr6Repz5g@mail.gmail.com>
	<CAJ_m3YDBY1OEWRrXZiCpVDX1eF=DVMw4rCojAv-Z59Fs5vPKRg@mail.gmail.com>
Date: Thu, 11 Oct 2012 16:32:13 -0700
Message-ID: 
 <CANpCgCM-G=QY3_61JnPxgTjhzMoPP3rORFTu-9nFBXsc=baZdQ@mail.gmail.com>
Subject: Re: pushing limits and possible corrupt db
From: Jason Konrad <jason@inboxq.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1

There was plenty of room left on the disk so I don't think that was
the problem.

I've managed to find this at the end of the .couch file.

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000390  01 00 00 00 6b ba 06 44  2d c7 74 3f 55 7e 40 74  |....k..D-.t?U~@t|
000003a0  1a 80 75 48 1c 83 68 0b  64 00 09 64 62 5f 68 65  |..uH..h.d..db_he|
000003b0  61 64 65 72 61 05 62 05  7d 07 b3 61 00 68 02 6e  |adera.b.}..a.h.n|
000003c0  05 00 7d c5 1a b1 27 68  02 62 00 92 e5 46 62 00  |..}...'h.b...Fb.|
000003d0  07 26 6d 68 02 6e 05 00  32 e5 1a b1 27 62 00 9a  |.&mh.n..2...'b..|
000003e0  0b b3 68 02 6e 05 00 6a  82 92 f3 0c 6a 61 00 64  |..h.n..j....ja.d|
000003f0  00 03 6e 69 6c 64 00 03  6e 69 6c 62 00 00 03 e8  |..nild..nilb....|
00000400

This looks like a valid header according to another post on the
mailing list which mentions the format "01 00 00 00 .... .db_header
.... 00 00 03 E8".

I have not tried to open the .couch file yet since I have to setup
another couchdb to use. Will be trying that soon.

On Thu, Oct 11, 2012 at 3:10 PM, Paul Davis <paul.joseph.davis@gmail.com> wrote:
> Definitely sounds like an emfile error. Could be that erlang
> translates that when opening a file but I'd have to check. I have seen
> issues with .couch files having issues when running out of disk space
> and the like. To recover, I would make a copy of your .couch file, and
> then start truncating it a bit at a time to try and find the last
> valid header that you can read from.
>
> On Thu, Oct 11, 2012 at 3:38 PM, Jason Konrad <jason@inboxq.com> wrote:
>> Today my Couch (1.0.1) took a turn for the worse. Couchdb seemed to be
>> stuck in some sort of error loop which rendered all databases useless.
>> I managed to isolate the problem to a specific database which was
>> causing the following error.
>>
>> [Thu, 11 Oct 2012 17:26:02 GMT] [error] [<0.595.0>] ** Generic server
>> <0.595.0> terminating
>> ** Last message in was {pread_iolist,170475046269}
>> ** When Server state == {file,{file_descriptor,prim_file,{#Port<0.2136>,23}},
>>                               0,170475073648}
>> ** Reason for termination ==
>> ** {{badmatch,{ok,<<99,111,109,104,2,110,5,0,186,181,74,98,39,104,2,98,0,0
>>                            ....SNIP...
>>                     106,97,0,100,0,3,110,105,108,100,0,3,110,105,108,98,0,0,
>>                     3,232>>}},
>>     [{couch_file,read_raw_iolist_int,3},
>>      {couch_file,handle_call,3},
>>      {gen_server,handle_msg,5},
>>      {proc_lib,init_p_do_apply,3}]}
>>
>> I stopped couchdb and moved the entire .couch file out of the database
>> directory and then started couchdb. After that couchdb appeared to be
>> normal with the other databases. The database in questions is ~159GB
>> with ~9M documents and each document has 2-10 attachments.
>>
>> I have recently been running up against some system_limit errors which
>> may have something to do with this although I have not seen any of
>> these errors today. I'm also trying to track down what system_limit
>> I'm hitting. If it was open files wouldn't it be a emfile error?
>>
>> ** {{badmatch,{error,system_limit}},
>>     [{couch_file,sync,1},
>>      {couch_db_updater,commit_data,2},
>>      {couch_db_updater,update_docs_int,5},
>>      {couch_db_updater,handle_info,2},
>>      {gen_server,handle_msg,5},
>>      {proc_lib,init_p_do_apply,3}]}
>>
>>
>> Any thoughts or suggestions would be appreciated.
>>
>> -Jason