incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luis Miguel Silva <luismiguelferreirasi...@gmail.com>
Subject Re: CouchDB crashing on me (while doing benchmark with 1 million documents)
Date Tue, 05 Apr 2011 15:37:27 GMT
Cliff,

Here is some more information:
- couchdb - Apache CouchDB 1.0.2
- CentOS 5.5 x64
- kernel 2.6.18-164.11.1.el5xen
- on a core 2 duo @ 2.66GHz with 4GB ram

Here is the script i used to generate my sample json:
[root@xkitten ~]# cat generate.pl
#!/usr/bin/perl


die "syntax: ./generate.pl <number of documents>\n" unless $#ARGV >= 0;

print "{\n";
print "\t\"docs\": [\n";
for($i = 0; $i < $ARGV[0]; $i++)
{
  print '   {"type": "node", "amemory": 14543, "cmemory": 49118,
"cproc": 16, "cpuload": 2.379, "netaddress": "10.40.130.146", "power":
"on", "primary_state": "UP", "secondary_state": "Unknown:sshd", "os":
"esxi4-stateless", "alias": "node01",  "vlans": "[vlan611]",
"variables": [ { "provision_status": 2 }, { "another_variable":
"something" } ], "aproc": 16, "vmcount": 6, "cpupercent":
14.8677794448612, "acores": 8, "ccores": 8, "threads": 16, "cpuclock":
2666, "numvms": 13, "hvtype": "esx", "htenabled": true },' . "\n";
}
print "\t]\n";
print "}\n";
[root@xkitten ~]#

time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &
time  curl -H "Content-Type: application/json" -d @docs.json -X POST
http://127.0.0.1:5984/benchmark/_bulk_docs &

Best,
Luis

On Tue, Apr 5, 2011 at 7:59 AM, Luis Miguel Silva
<luismiguelferreirasilva@gmail.com> wrote:
> It is version 1.0.1 (i think - i can't connect to my server right now, but it's basically
the latest stable version available for download in the official web site), and i'm running
on CentOS 5.5 x64.
>
> Do you want the steps i used to reproduce this?
>
> p.s. i also tried submitting 1 million documents to the bulk_docs API (from one process)
and after 33 minutes the process still hadn't finished so i just killed it...
> p.p.s. as for the test that crashed couchdb, i actually tried it two or three times (after
restarting couchDB and starting with a brand new / clean DB)...
>
> Thanks,
> Luis Miguel Silva
>
> On Apr 5, 2011, at 2:51 AM, Cliff Williams <cliffywills@aol.com> wrote:
>
>> Luis,
>>
>> good morning.
>>
>> What platform (linux/windows) and version of couchdb are you running??
>>
>> best regards
>>
>> Cliff
>>
>> On 05/04/11 06:49, Luis Miguel Silva wrote:
>>> Dear all,
>>>
>>> I'm doing some benchmarks on CouchDB and it seems i'm making it crash
>>> when i try to create 1 million parallel documents, using 10 processes,
>>> submitting 100000K documents each (using the bulk_docs API).
>>> Is there anything i can debug to figure out what is going on? The
>>> server just dies and the logs do not show anything special...
>>>
>>> The only thing i see in the client side is that the server just
>>> unexpectedly closed the connection and did not return anything...
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>> curl: (52) Empty reply from server
>>>
>>> real    1m20.955s
>>> user    0m0.180s
>>> sys     0m0.148s
>>>
>>> real    1m21.034s
>>> user    0m0.140s
>>> sys     0m0.064s
>>>
>>> real    1m21.036s
>>> user    0m0.144s
>>> sys     0m0.088s
>>>
>>> real    1m21.035s
>>> user    0m0.192s
>>> sys     0m0.124s
>>>
>>> real    1m21.041s
>>> user    0m0.184s
>>> sys     0m0.168s
>>>
>>> real    1m21.040s
>>> user    0m0.108s
>>> sys     0m0.136s
>>>
>>> real    1m21.048s
>>> user    0m0.136s
>>> sys     0m0.092s
>>>
>>> real    1m21.050s
>>> user    0m0.136s
>>> sys     0m0.132s
>>>
>>> real    1m21.049s
>>> user    0m0.112s
>>> sys     0m0.116s
>>>
>>> real    1m21.059s
>>> user    0m0.132s
>>> sys     0m0.092s
>>>
>>> [root@xkitten ~]#
>>>
>>> Any thoughts?
>>>
>>> p.s. does anybody else have some interesting benchmarks they can hare with me?
>>>
>>> Thank you!
>>> Luis
>>>
>

Mime
View raw message