Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1257)
Subject: Re: {error,emfile} on CouchDB 1.2.x
From: Jan Lehnardt <jan@apache.org>
In-Reply-To: 
 <CAAL6JQhnXXZiCfu5hnibgjdvAbwwrQDTzkXxH-fm_5nQEF-GsQ@mail.gmail.com>
Date: Sun, 18 Mar 2012 21:48:50 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <5D7F6BB2-EBFE-40F2-A6B1-33BA8A5972DF@apache.org>
References: <4F6624AA.7050202@gmail.com>
 <CAAL6JQg+u5sjOuYobfxKM+AnC=mWyQ_GDSXcwXcB1t5V60xYcQ@mail.gmail.com>
 <B540EFE4-E585-48F3-B8A5-D324D6A9E7E0@apache.org>
 <CAAL6JQhnXXZiCfu5hnibgjdvAbwwrQDTzkXxH-fm_5nQEF-GsQ@mail.gmail.com>
To: dev@couchdb.apache.org


On Mar 18, 2012, at 21:46 , Randall Leeds wrote:

> On Sun, Mar 18, 2012 at 13:39, Jan Lehnardt <jan@apache.org> wrote:
>=20
>>=20
>> On Mar 18, 2012, at 21:28 , Randall Leeds wrote:
>>=20
>>> On Sun, Mar 18, 2012 at 11:08, Stefan K=F6gl <koeglstefan@gmail.com>
>> wrote:
>>>=20
>>>> Hi,
>>>>=20
>>>> Another thing I noticed during my tests of CouchDB 1.2.x. I =
redirected
>>>> live traffic to the instance and after a rather short time, =
requests
>>>> were failing with the following information in the logs:
>>>>=20
>>>>=20
>>>> [Sun, 18 Mar 2012 16:39:24 GMT] [error] [<0.27554.2>]
>>>> {error_report,<0.31.0>,
>>>>                                  {<0.27554.2>,std_error,
>>>>                                   [{application,mochiweb},
>>>>                                    "Accept failed error",
>>>>                                    "{error,emfile}"]}}
>>>> [Sun, 18 Mar 2012 16:39:24 GMT] [error] [<0.27554.2>]
>>>> {error_report,<0.31.0>,
>>>>                        {<0.27554.2>,crash_report,
>>>>                         [[{initial_call,
>>>>                               {mochiweb_acceptor,init,
>>>>                                   ['Argument__1','Argument__2',
>>>>                                    'Argument__3']}},
>>>>                           {pid,<0.27554.2>},
>>>>                           {registered_name,[]},
>>>>                           {error_info,
>>>>                               {exit,
>>>>                                   {error,accept_failed},
>>>>                                   [{mochiweb_acceptor,init,3},
>>>>                                    {proc_lib,init_p_do_apply,3}]}},
>>>>                           {ancestors,
>>>>                               =
[couch_httpd,couch_secondary_services,
>>>>                                couch_server_sup,<0.32.0>]},
>>>>                           {messages,[]},
>>>>                           {links,[<0.129.0>]},
>>>>                           {dictionary,[]},
>>>>                           {trap_exit,false},
>>>>                           {status,running},
>>>>                           {heap_size,233},
>>>>                           {stack_size,24},
>>>>                           {reductions,244}],
>>>>                          []]}}
>>>>=20
>>>>=20
>>>> I think "emfile" means that CouchDB (or mochiweb?) couldn't open =
any
>>>> more files / connections. I've set the (hard and soft) nofile limit =
for
>>>> user couchdb to 4096, but didn't raise the ERL_MAX_PORTS =
accordingly.
>>>> Anyway, as soon as the error occured, CouchDB started writing most =
of my
>>>> view files from scratch, rendering the instance unusable.
>>>>=20
>>>> I'd expect CouchDB to fail more gracefully when the maximum number =
of
>>>> open files is reached. Is this a bug or expected behaviour?
>>>>=20
>>>=20
>>> Looks like a bug. Whenever there's a problem opening a view file,
>>> couch_view tries to delete it. Clearly, this is not the right course =
of
>>> action when the problem is due to emfile.
>>=20
>> This looks rather serious. I opened a JIRA:
>>=20
>> https://issues.apache.org/jira/browse/COUCHDB-1445
>>=20
>> And started collecting the info. Bob N's message came in in the =
meantime
>> and I agree, we should see if there's more cases where we need to be
>> careful.
>>=20
>> Also, I'd consider this blocking for 1.2.0.
>>=20
>> Anyone who can pitch in with their expertise is more than welcome! :)
>>=20
>=20
> Assigned to me. Patch forthcoming. Agree in should block 1.2.0, =
especially
> because upgrades are the sort of things where bad packaging downstream
> might cause custom ERL_MAX_PORTS settings to be overwritten and we =
wouldn't
> want anyone's production to have its views erased needlessly.

Thanks for taking this on Randall!

Cheers
Jan
--=20

>=20
> -Randall
>=20
>=20
>>=20
>> Cheers
>> Jan
>> --
>>=20
>>=20
>>>=20
>>> Here's a patch that I propose might fix it. I'd like to hear from =
another
>>> dev on this, or if there's a better way we should bail out.
>>>=20
>>> diff --git a/src/couchdb/couch_view_group.erl
>>> b/src/couchdb/couch_view_group.erl
>>> index 97fc512..ab075bd 100644
>>> --- a/src/couchdb/couch_view_group.erl
>>> +++ b/src/couchdb/couch_view_group.erl
>>> @@ -469,6 +469,10 @@ open_index_file(RootDir, DbName, GroupSig) ->
>>>    case couch_file:open(FileName) of
>>>    {ok, Fd}        -> {ok, Fd};
>>>    {error, enoent} -> couch_file:open(FileName, [create]);
>>> +    {error, emfile} ->
>>> +        ?LOG_ERROR("Could not open file for view index: max open =
files
>>> reached. "
>>> +                   "Raise ERL_MAX_PORTS or system limits.", []),
>>> +        throw({error, emfile});
>>>    Error           -> Error
>>>    end.
>>=20
>>=20