couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Stockton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
Date Wed, 24 Aug 2011 21:04:29 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090484#comment-13090484
] 

Chris Stockton commented on COUCHDB-1257:
-----------------------------------------

Adam it appears this error is a bit more serious then original understood. While this has
been happening we were having a similar error with views no longer returning results and just
timing out. It seems each reduce over flow error results in a couchjs process remaining open
forever.


[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
14


<induce error 10 times>


[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
24


This ultimately leads to couch becoming unstable.

> Reduction detection in share/server/views.js runReduce to primitive for some use cases
and overall reduce architecture
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: COUCHDB-1257
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1257
>             Project: CouchDB
>          Issue Type: Bug
>          Components: JavaScript View Server
>    Affects Versions: 1.1
>         Environment: Architecture independent
>            Reporter: Chris Stockton
>            Priority: Minor
>              Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to the protocol
in which couchdb actually talks to the view server (server/main.js), the second part as discussed
here [1] in short is the growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a reduce
function for every doc called, a separate read line is delegated a reset call. I do not know
the internals of couchdb but it seems like it could be a performance improvement if on a rereduce
call, as it appears in the server Loop function every time reduce or rereduce is called the
state is first reset with "reset", to make a minor re-factor and reset the state within the
Views dispatcher itself. This saves some likely blocking io back and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection done, I think
that it should be changed to require a sampling of data, the easiest fix is to simply change
(the very arbitrary) [2] into a large length, such as * 10 or something, but the upper bounds
will always be reached for some data set. A new algorithm could take in consideration the
total reduce calls thus far, as well as the initial size of the very first reduce call, to
see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count of rereduce
calls thus far, or give some kind of handle where this logic could be reliably kept track
of inside of main.js, as it stands now your only hope would be to use the reduce function
itself as a lookup, which is not safe under all edge cases. With this information you can
keep the initial rereduce calls result size, and the total number of calls, to make sure that
aside from initial aggregation or structure initialization the structure is remaining constant.
> There would be several other routes to go to fix this, but I think it is valid to be
considered. I could volunteer my time to write a patch if a consensus could be reached on
behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2]     if (State.query_config && State.query_config.reduce_limit &&
> [2]          reduce_length > 200 && ((reduce_length * 2) > State.line_length))
{

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message