Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of vkuznet@gmail.com designates
 209.85.216.199 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type:mime-version:subject
         :date:references:x-mailer;
        b=pTv8OmOVsibUkHEOfVjVyY2t16SwMN+wa4P5nKLtP1GGg9AlrnZTzKbYPEqNeqidKR
         6sprJZ6bwNNSmXtlgbODyhhQmSswN2SMbW7MRSWrZf6wyAP+7+JmJ4aGYPvb6Y44RTwo
         yIYIVhC1mGxH4xI+nuaYHaiAs1HKgDLAs6jcI=
Message-Id: <C3EC33CF-1671-4E11-ACC1-A69ED7620703@gmail.com>
From: Valentin Kuznetsov <vkuznet@gmail.com>
To: user@couchdb.apache.org
In-Reply-To: <e282921e0907031146l67e4bedeu2e66397a37eb569c@mail.gmail.com>
Content-Type: multipart/signed; boundary=Apple-Mail-38--91659200; micalg=sha1;
 protocol="application/pkcs7-signature"
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: transient errors with couch DB.
Date: Fri, 3 Jul 2009 15:56:36 -0400
References: <0E4628F2-951C-4BCB-9E4C-E7F2164E0DFC@gmail.com>
 <e282921e0907031146l67e4bedeu2e66397a37eb569c@mail.gmail.com>

--Apple-Mail-38--91659200
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Chris,
thank you for suggestions. The problem is that we tried out all of  
those and encounter other problems. If I inject 64K rows at once, then  
memory consumption as well as CPU got high. Moreover the view after  
that takes literally more then 10 minutes to compute. We're working in  
environment where 1000 simultaneous requests are possible. Our  
preliminary tests shown that view computation grows significantly with  
number of inserted docs. So what I present we found optimal in our  
situation.

Regarding sockets. It well may be and I'll be willing to explore this.  
But the code example I posted did sequential insertion of docs.  
Basically open HTTP connection, insert docs, close it.
Valentin

On Jul 3, 2009, at ,Jul 3, 2:46 PM, Chris Anderson wrote:

> On Thu, Jul 2, 2009 at 7:35 AM, Valentin  
> Kuznetsov<vkuznet@gmail.com> wrote:
>> Hi,
>> I'm developing application based on couch db and time to time  
>> experience
>> different error messages and crashes with couch, version 0.9.0. We  
>> have a
>> large documents, e.g. 64K, which split into 1K chunks (rows) for bulk
>> injection. Each time I inject new chunk I also invoke 1 GET request  
>> w/ 1 row
>> to force view to index data.
>
> You should not be querying the view on every insert, once every few
> thousand docs should be fine...
>
> I'd maybe go ahead and not break up the 64Kb docs into smaller piece,
> Couch should be able to handle docs of that size just fine.
>
> A quick google reveals that einval has got something to do with
> opening sockets. My guess is that you are opening too many at once for
> your OS. You may be able to do some linux config to raise the limit,
> but I'd start by simplifying your insert loop.
>
>
>>
>> So here are error messages:
>>
>> [error] [<0.12304.2>] Uncaught error in HTTP request: {exit,normal}
>>
>> [error] [<0.30955.5>] {error_report,<0.23.0>,
>>    {<0.30955.5>,crash_report,
>>     [[{initial_call,{mochiweb_socket_server,acceptor_loop, 
>> ['Argument__1']}},
>>       {pid,<0.30955.5>},
>>       {registered_name,[]},
>>       {error_info,
>>           {error,
>>               {case_clause,{error,einval}},
>>               [{mochiweb_request,get,2},
>>                {couch_httpd,handle_request,4},
>>                {mochiweb_http,headers,5},
>>                {proc_lib,init_p_do_apply,3}]}},
>>       {ancestors,
>>            
>> [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
>>       {messages,[]},
>>       {links,[<0.53.0>,#Port<0.1735>]},
>>       {dictionary,[{mochiweb_request_qs,[{"limit","1"}]}]},
>>       {trap_exit,false},
>>       {status,running},
>>       {heap_size,6765},
>>       {stack_size,24},
>>       {reductions,3888}],
>>      []]}}
>> [error] [<0.53.0>] {error_report,<0.23.0>,
>>    {<0.53.0>,std_error,
>>
>> {mochiweb_socket_server,235,{child_error,{case_clause, 
>> {error,einval}}}}}}
>>
>> =ERROR REPORT==== 2-Jul-2009::10:21:33 ===
>> {mochiweb_socket_server,235,{child_error,{case_clause, 
>> {error,einval}}}}
>>
>> So, I made a stand-alone program and to my surprise the errors are
>> transient. I re-run my stand-alone program with the same doc  
>> several times
>> and I can either inject all data just fine or get errors shown  
>> above in no
>> particular order (I mean I can see 1 or more errors and none or  
>> either of
>> those).
>>
>> I would appreciate any help with this. The program is attached in a  
>> form of
>> tar archive. I added data file we usually deal with and two python  
>> files
>> which we use. To run a test untar archive and do the following
>>
>> export PYTHONPATH=$PWD
>> python couch_test.py
>>
>>
>>
>>
>>
>> Thank you,
>> Valentin
>>
>>
>>
>
>
>
> -- 
> Chris Anderson
> http://jchrisa.net
> http://couch.io


--Apple-Mail-38--91659200
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIDhTCCA4Ew
ggJpoAMCAQICAmT/MA0GCSqGSIb3DQEBBQUAMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJ
kiaJk/IsZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAU
BgNVBAMTDURPRUdyaWRzIENBIDEwHhcNMDgwODE5MTYyNzI2WhcNMDkwODE5MTYyNzI2WjBkMRMw
EQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIZG9lZ3JpZHMxDzANBgNVBAsTBlBl
b3BsZTEiMCAGA1UEAxMZVmFsZW50aW4gS3V6bmV0c292IDM1NzY4MzCBnzANBgkqhkiG9w0BAQEF
AAOBjQAwgYkCgYEA4iFFg1Vubs0frG4NaLheE4xA4zLEvZt6N6eCnqLK6IbiR4XoE0W7gRNs12G5
jNlTp/gyHrt5ckXQ7JLcz0L9GmKDUGXezSVqGqri62Ek5G3NIR8XhpHS1pQyUbP2mZCvkh5hU09x
2sXsBNhPLhqg8EHCzjL33RQe1uXzVNFJ1SMCAwEAAaOBuzCBuDARBglghkgBhvhCAQEEBAMCBeAw
DgYDVR0PAQH/BAQDAgTwMBgGA1UdIAQRMA8wDQYLKoZIhvdMAwcBAgowOgYDVR0fBDMwMTAvoC2g
K4YpaHR0cDovL3BraTEuZG9lZ3JpZHMub3JnL0NSTC8xYzNmMmNhOC5jcmwwHAYDVR0RBBUwE4ER
dmt1em5ldEBnbWFpbC5jb20wHwYDVR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZI
hvcNAQEFBQADggEBABIX88qlO0z/xNPeSAVa0XG8fLqu9M9Qt5yIqUhaG+Yp2JDDzlHghqaNcXmR
C6JehHUiEpNKU7HdI1xkU1PiJbBlTct+r+waG3J6KJwT3E+V6HmDvvNaQXzVe0WMr9ldYkDbjI1x
ekqzpDH8RYEN7MmVU3AZQODJKfqr7PdXHJIJtktFvdapUj/gwNLYWzjcu4ZFNfXORfqgIns7RE7Z
MT8+PPnOqu075snLNJxXGmaMPua3te/ZYvB3SQc69pdTQxfPHzl0h7KwKP3iCe5Z/UkVjebK7eeF
Yn2kPKbRO24RPLhbkl/OXdNOK0p8C7DQlpXebID5V2Pei8H5LMXDgKUxggJ5MIICdQIBATBvMGkx
EzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEgMB4GA1UECxMX
Q2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDECAmT/MAkGBSsO
AwIaBQCgggFgMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDcw
MzE5NTYzN1owIwYJKoZIhvcNAQkEMRYEFOdtCljH3a0O57jxRSzFG6hjs31xMH4GCSsGAQQBgjcQ
BDFxMG8waTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAw
HgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIC
ZP8wgYAGCyqGSIb3DQEJEAILMXGgbzBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy
LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD
Ew1ET0VHcmlkcyBDQSAxAgJk/zANBgkqhkiG9w0BAQEFAASBgMWSL2Ld5wRyB0gHuQuR3tK3C0R/
KBjW4HsibUdh26sHrJV33MYjCtIUrAtgfm7Lw1CCKULdfz49MrnajqY5TXMbFO8MRkOPblA3ggGw
xRsulzKH5Grzb2WbuDlgt7ZwnimesEEvfKbds38n8ikdczr4lXkyf+uUStuh/cLHNi06AAAAAAAA

--Apple-Mail-38--91659200--