incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Cheng <johnlich...@gmail.com>
Subject Re: CouchDB performance
Date Tue, 16 Aug 2011 17:03:29 GMT
On Thu, Aug 11, 2011 at 12:29 PM, John Cheng <johnlicheng@gmail.com> wrote:

> On Thu, Aug 11, 2011 at 11:36 AM, John Cheng <johnlicheng@gmail.com>wrote:
>
>> On Thu, Aug 11, 2011 at 11:02 AM, Adam Kocoloski <kocolosk@apache.org>wrote:
>>
>>> Hi John, first thing I'd point out is that Filipe and Paul have done
>>> quite a lot of work on the JSON processor since 1.0.2 was released.  Trunk
>>> contains a JSON processor that calls out to YAJL instead of relying on pure
>>> Erlang code.  The improvement for large document bodies is pretty
>>> substantial.  Also, processing MBs of JSON is going to be CPU-bound in
>>> 1.0.2.  CouchDB makes good use of multicore servers to distribute some of
>>> the load if you throw concurrent requests at it.  Hope that helps,
>>>
>>> Adam
>>>
>>
>> It does help here. I'll see if I can get compile the latest version of
>> CouchDB on Ubuntu 10 LTS. It's been an adventure trying to compile CouchDB
>> so far!
>>
>
>
> Just compiled CouchDB 1.1.0
>
> Testing with doc target size = "100 kB", # of properties = "10", updates =
> "500", numDocs = "100"
> each document is approximately 100.1 kB
> Using increment handler function to perform update
> 500 updates in 42.1960351467 seconds, 11.8494545343 ops/second
>
> Now the updates dropped from 23 updates/sec to 11.84 updates/second
>
> Even inserts dropped to 24 inserts/second. Accroding to iostat, CPU was
> mostly idle (~85 to 92%) and I'm only getting 300 kB writes per second.
>
> I guess this could be a problem with httplib2. Is there any configuration
> or compile time options I may have missed that could cause such a big drop
> in performance?
>
> --
> ---
> John L Cheng
>

Just to follow up on this. I've pushed a test case written in Scala to
Github (https://github.com/jlcheng/couchdb-test). With this test case, I am
using Apache HttpClient to perform testing. Since I am more familiar with
Java, I was able to optimize the test case more. With Scala and CouchDB
0.10.0 (default install from Ubuntu 10.04 LTS), I am seeing 94 inserts per
second, document size of 100k (using batch=ok). I think this pretty close to
being optimized.

However, going to a custom compiled CouchDB 1.1.0, I am seeing a mere 20
inserts per second. I also see disk activity drop from ~7mb writes per
second to ~2mb/sec. There must be something wrong with the way I have it
compiled or configured. Any ideas where I should start?


-- 
---
John L Cheng

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message