incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: view building batch sizes
Date Thu, 09 Dec 2010 15:47:07 GMT

On 9 Dec 2010, at 15:37, Paul Davis wrote:

> On Thu, Dec 9, 2010 at 7:51 AM, Jan Lehnardt <jan@apache.org> wrote:
>> Hi Huw,
>> 
>> 
>> On 9 Dec 2010, at 13:32, Huw Selley wrote:
>> 
>>> Hi,
>>> 
>>> I read on http://guide.couchdb.org/draft/performance.html that
>>> 
>>> "Views load a batch of updates from disk, pass them through the view engine,
and then write the view rows out. Each batch is a few hundred documents, so the writer can
take advantage of the bulk efficiencies we see in the next section."
>>> 
>>> Is there a method to change the batch size? I would like to try measure the impact
of using smaller and larger batches.
>> 
>> Thanks for helping to profile things. You may want to take this to
>> dev@couchdb.apache.org as it is the development-related mailing list.
>> 
>> For tuning these values, see src/couchdb/couch_view_updater.erl
>> 
>> The `update()` function has these lines:
>> 
>>    {ok, MapQueue} = couch_work_queue:new(100000, 500),
>>    {ok, WriteQueue} = couch_work_queue:new(100000, 500),
>> 
>> They set up a queue for mapping and writing each. The parameters are
>> 
>>    couch_work_queue:new(MaxSize, MaxItems)
>> 
>> If either maximum is hit, the queue is deemed full.
>> 
>> Note: This is from about 30 seconds of looking at the source, so I
>> might miss a subtlety or three.
>> 
>> Cheers
>> Jan
>> --
>> 
>> 
>> 
> 
> The only real subtlety is that we don't wait for a minimum amount to
> be inserted into the queue. Playing with larger or smaller queues on
> either side might be an interesting bit. Also, for testing it might
> not be a bad idea to add config values for these values.


Good thinking, I made a patch:

  https://github.com/janl/couchdb/commit/547691a9f4b9895086f2763af84e1cc459e4d72c

Branch:

 https://github.com/janl/couchdb/tree/config-view-batches

"Compiles for me".

To make this proper, we probably want to move the lookups into 
couch_view_group:init/3 and pass the values down, but it should
be ok as is.

Cheers
Jan
-- 


Mime
View raw message