couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <rnew...@apache.org>
Subject Re: Performance issue with changes API
Date Wed, 12 Oct 2011 15:33:36 GMT
ooh, my math is way off, ignore ;)

On 12 October 2011 16:32, Robert Newson <rnewson@apache.org> wrote:
> The 3.5M row response is not formed in memory. :) It's done line by line.
>
> that said, that's almost 2000 rows per second, which doesn't sound
> that bad to me.
>
> B.
>
> On 12 October 2011 16:26, Matt Goodall <matt.goodall@gmail.com> wrote:
>> On 12 October 2011 14:22, Arnaud Bailly <arnaud.oqube@gmail.com> wrote:
>>> Hello,
>>> We have started experimenting with CouchDb as our backend, being especially
>>> interested with the changes API, and we ran into performances issues.
>>> We have a DB containing aournd 3.5M docs, each about 10K in size. Running
>>> the following query on the database :
>>>
>>> http://192.168.1.166:5984/infowarehouse/_changes?since=0
>>>
>>> takes about 30minutes on a 4-core, Windows 7 box, which seems rather high.
>>>
>>> Is this expected ?  Are there any bench available on this API ?
>>
>> I'm not too surprised - CouchDB is probably building a massive JSON
>> changes response containing 3.5M items ;-). Instead you should use the
>> since=<start> and limit=<batch-size> args together to get the items in
>> sensibly-sized batches, ending when you see no more items in the
>> response.
>>
>> Alternatively, you might be able to use feed=continuous with timeout=0
>> to stream the changes as fast as possible. The timeout=0 arg is just
>> there to shutdown the changes feed as soon as you've seen everything.
>> My laptop takes about 50s to stream about 1M changes using this
>> technique (sending the output to /dev/null).
>>
>> - Matt
>>
>

Mime
View raw message