couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kuznetsov <vkuz...@gmail.com>
Subject Re: transient errors with couch DB.
Date Fri, 03 Jul 2009 19:56:36 GMT
Chris,
thank you for suggestions. The problem is that we tried out all of  
those and encounter other problems. If I inject 64K rows at once, then  
memory consumption as well as CPU got high. Moreover the view after  
that takes literally more then 10 minutes to compute. We're working in  
environment where 1000 simultaneous requests are possible. Our  
preliminary tests shown that view computation grows significantly with  
number of inserted docs. So what I present we found optimal in our  
situation.

Regarding sockets. It well may be and I'll be willing to explore this.  
But the code example I posted did sequential insertion of docs.  
Basically open HTTP connection, insert docs, close it.
Valentin

On Jul 3, 2009, at ,Jul 3, 2:46 PM, Chris Anderson wrote:

> On Thu, Jul 2, 2009 at 7:35 AM, Valentin  
> Kuznetsov<vkuznet@gmail.com> wrote:
>> Hi,
>> I'm developing application based on couch db and time to time  
>> experience
>> different error messages and crashes with couch, version 0.9.0. We  
>> have a
>> large documents, e.g. 64K, which split into 1K chunks (rows) for bulk
>> injection. Each time I inject new chunk I also invoke 1 GET request  
>> w/ 1 row
>> to force view to index data.
>
> You should not be querying the view on every insert, once every few
> thousand docs should be fine...
>
> I'd maybe go ahead and not break up the 64Kb docs into smaller piece,
> Couch should be able to handle docs of that size just fine.
>
> A quick google reveals that einval has got something to do with
> opening sockets. My guess is that you are opening too many at once for
> your OS. You may be able to do some linux config to raise the limit,
> but I'd start by simplifying your insert loop.
>
>
>>
>> So here are error messages:
>>
>> [error] [<0.12304.2>] Uncaught error in HTTP request: {exit,normal}
>>
>> [error] [<0.30955.5>] {error_report,<0.23.0>,
>>    {<0.30955.5>,crash_report,
>>     [[{initial_call,{mochiweb_socket_server,acceptor_loop, 
>> ['Argument__1']}},
>>       {pid,<0.30955.5>},
>>       {registered_name,[]},
>>       {error_info,
>>           {error,
>>               {case_clause,{error,einval}},
>>               [{mochiweb_request,get,2},
>>                {couch_httpd,handle_request,4},
>>                {mochiweb_http,headers,5},
>>                {proc_lib,init_p_do_apply,3}]}},
>>       {ancestors,
>>            
>> [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
>>       {messages,[]},
>>       {links,[<0.53.0>,#Port<0.1735>]},
>>       {dictionary,[{mochiweb_request_qs,[{"limit","1"}]}]},
>>       {trap_exit,false},
>>       {status,running},
>>       {heap_size,6765},
>>       {stack_size,24},
>>       {reductions,3888}],
>>      []]}}
>> [error] [<0.53.0>] {error_report,<0.23.0>,
>>    {<0.53.0>,std_error,
>>
>> {mochiweb_socket_server,235,{child_error,{case_clause, 
>> {error,einval}}}}}}
>>
>> =ERROR REPORT==== 2-Jul-2009::10:21:33 ===
>> {mochiweb_socket_server,235,{child_error,{case_clause, 
>> {error,einval}}}}
>>
>> So, I made a stand-alone program and to my surprise the errors are
>> transient. I re-run my stand-alone program with the same doc  
>> several times
>> and I can either inject all data just fine or get errors shown  
>> above in no
>> particular order (I mean I can see 1 or more errors and none or  
>> either of
>> those).
>>
>> I would appreciate any help with this. The program is attached in a  
>> form of
>> tar archive. I added data file we usually deal with and two python  
>> files
>> which we use. To run a test untar archive and do the following
>>
>> export PYTHONPATH=$PWD
>> python couch_test.py
>>
>>
>>
>>
>>
>> Thank you,
>> Valentin
>>
>>
>>
>
>
>
> -- 
> Chris Anderson
> http://jchrisa.net
> http://couch.io


Mime
View raw message