Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: couchdb-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of nickretallack@gmail.com
 designates 209.85.134.185 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=vifM61KajgvXxxwYxFXjMO9SzJAsORGkEaF5arrsA7mLqnsrCZfZ7tFgMRUvpHVXSU
         q71lsoAYFBZuO07ZttdEzFRYVnt6c2vrZbjqjpVfVdU3JjnWILOBLBiJUUZXTZNPfanp
         BV59NufIsD/GNM+0lMQlZT9XVpop3eoUpMbyw=
Message-ID: <66e809970808182327m4ede024bw131f3c13a19ef31c@mail.gmail.com>
Date: Mon, 18 Aug 2008 23:27:12 -0700
From: "Nicholas Retallack" <nickretallack@gmail.com>
To: couchdb-user@incubator.apache.org
Subject: Reduce is Really Slow!
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_27139_25407029.1219127232296"

------=_Part_27139_25407029.1219127232296
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

You may have noticed my old post talking about reduce's weird behavior.
Well, I just tried out a reduce technique on a relatively large database,
and it seems quirks are the least of our problems.

This is the view I used: http://www.friendpaste.com/2AHz3ahr

My database is 78.3MB and contains 77604 documents, about 34000 of which
satisfy doc.Type == 'offer'.  I will do some queries with &count=1 to
minimize download time as a factor.

A normal query on this view takes 4.5 seconds.  That's pretty bad, since I
wanted to use this for a web interface.  If I add group=true&group_level=1
to my query string, it jumps up to 25 seconds!  If I remove the reduce
method from my view, queries take only 44 milliseconds.

I could not have predicted this big a performance problem just from using
reduce.  This makes it pretty unusable as a web interface.  In fact, I might
be better off doing the reduce operation myself in the application layer,
which is not very awesome at all.

Poking further, I see a huge performance problem even with the simplest of
reduce functions, like function(keys,values){return values} for example.  I
couldn't get a benchmark on this one because it took so long to run that
couchdb timed out and killed it.
> curl -X GET http://localhost:5984/clickfund/_view/offers/index
{"error":"error","reason":"{{nocatch,{map_process_error,\"map function timed
out\"}},\n [{couch_query_servers,readline,2},\n
 {couch_query_servers,read_json,1},\n  {couch_query_servers,prompt,2},\n
 {couch_query_servers,'-rereduce\/3-fun-0-',3},\n  {lists,zipwith,3},\n
 {couch_query_servers,rereduce,3},\n
 {couch_view,'-init_group\/4-fun-0-',4},\n
 {couch_btree,'-write_node\/3-lc$^0\/1-0-',3}]}"}

I am using couchdbx 0.8.0  I have been told that 0.8.1 improves on
javascript views, but I was unable to get it working on my mac.  It compiles
and runs, but when I try to connect to it I am refused.  This is strange,
because Couchdbx works fine when I run it on the same port.

> couchdb
Apache CouchDB 0.9.0a686685-incubating (LogLevel=info)
Apache CouchDB is starting.

Config Info /usr/local/etc/couchdb/couch.ini:
CurrentWorkingDir=/Users/nick/hooraycouch
DbRootDir=/usr/local/var/lib/couchdb
BindAddress="127.0.0.1"
Port="5984"
DocumentRoot=/usr/local/share/couchdb/www
LogFile=/usr/local/var/log/couchdb/couch.log
UtilDriverDir=/usr/local/lib/couchdb/erlang/lib/couch-0.9.0a686685-incubating/priv/lib
DbUpdateNotificationProcesses=
FullTextSearchQueryServer=
javascript=/usr/local/bin/couchjs /usr/local/share/couchdb/server/main.js

(on another terminal)
> curl -X GET http://localhost:5984/_all_dbs
curl: (7) couldn't connect to host

------=_Part_27139_25407029.1219127232296--