incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Felix Girard <jeanfel...@icloud.com>
Subject Filter Performance
Date Wed, 21 Aug 2013 14:37:26 GMT
Hi Couchdb users,
	
Here is my situation:

	I'm using Couchdb 1.3.1 for my web app.  Each user do a long-poll to _changes to exchange
messages and other informations on a shared database.  A user can load a lot of docs at once
using a server-side file loader.   I set up a FILTER to use on _changes to only include "message"
doc.  It works, but it's much slower then expected.  All my views / filters are written in
Erlang.


For a simple database with *only* 56K documents, here is the performance I see on localhost:

Build a view indexing all docs (from scratch)	 					:  ~ 6 seconds
Query _changes with no filter, include_docs=true, since=0		:  ~ 10 seconds  (its basically
downloading all the docs from the database)
Query _changes with a simple filter, include_docs=true, since=0	:  ~ 47 seconds  (and return
no doc)

It grows linearly with doc count.  112 K docs: 

Build a view indexing all docs (from scratch)	 					: ~10 s 
Query _changes with no filter, include_docs=true, since=0 		: ~ 22s
Query _changes with a simple filter, include_docs=true, since=0	: ~ 95s 


My filter is fairly simple:

"fun({Doc}, {Req}) -> DocType = proplists:get_value(<<\"type__\">>, Doc), case
DocType of <<\"message\">> -> true;  _ -> false end end."
 

The view is a bit more complicated:

"fun({Doc}) -> DocTrashed = proplists:get_value(<<"trashed__">>, Doc), case
DocTrashed of true -> ok; _ -> DocType = proplists:get_value(<<"type__">>,
Doc),  Emit([proplists:get_value(<<"created_at__">>, Doc)], null); _ -> ok
end  end."


My filter used to be written in JS and I did see a big change using Erlang.  For 56K, it was
2min +.

I  do take avantage of the since=the_last_seq to minimize the time to wait... But there are
situations where let say 2 users are connected (both listening to _changes of the same db)
and one load a file.  The other user will then "hang" for while, not being able to receive
messages.  


Is it the expected performance?  Any performance tweak or config to do ?

Thanks.





Mime
View raw message