incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Trümper <daniel.truem...@googlemail.com>
Subject Re: Updating views while inserting documents
Date Mon, 13 Jul 2009 21:19:01 GMT

>> - calling the view takes hours! at the moment: _1_ document every 5  
>> seconds!
> Huh, do you mean that the Status item showing the progress of the  
> indexer only updates once every 5 seconds?  Or that downloading each  
> row of the view takes 5 seconds.
Obviously an inaccurate comment of mine. What I was trying to say is  
that in the status item showing the progress of the indexer shows,  
that only every 5 seconds one document is being processed. I am having  
an idea that it might be caused by a map function emitting to much  
data into the key, but I am not sure. Will have more time to  
investigate on this tomorrow!


> Do you have a reduce function for these views?
Yes, I do, but until I read

> http://mail-archives.apache.org/mod_mbox/couchdb-dev/200907.mbox/%3c4A4E7F53.7010406@krampe.se%3e
I did not think that they are be the problem. But now I think they  
could be :)

Essentially I am trying to create a view that should return all unique  
values of a given field across documents. I don't know what fields the  
user will store so they are emitted. Here is my map function:

---snip-map--
function(doc) {
   for( var field in doc ) {
     if( field.indexOf('_') != 0 ) {
       emit( field, doc[field] );
     }
   }
}
---snip-map--

and the reduce:

---snip-reduce--
function(keys,values,rereduce) {
   var o = new Object();
   var i, e;
   for (i = 0; e = values[i]; i++) {
     if (rereduce) {
       for(j=0; h=e[j]; j++){ o[h] = 1; }
     } else {
       o[e] = 1;
     }
   };
   var a = new Array();
   for (e in o) { a.push (e) };
   return a;
}
---snip-reduce--

Ok, now I am waiting for the punishment about my bad reduce and  
whatever, but this is my first project with CouchDB, so be kind :)

The problem now probably is caused by the user who added quite large  
data into a field. The library I am writing was originally only  
intended to store things like Strings and the like, but now there are  
large objects in the fields value. Now since they are emitted in the  
map and reduced and rereduced in the reduce function they most  
probably cause the extremely large view file and the slow performance.  
Or any other guesses? I will try to fix this tomorrow!

> At the time we assumed it was an issue with the reduce function "not  
> reducing enough", but if you're finding this same problem with a map- 
> only function then it really makes me wonder if we've got a more  
> serious issue.  Best,
So, no, I guess it's not a serious issue, but my design document.

> Adam
Daniel

Mime
View raw message