couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Leeds <randall.le...@gmail.com>
Subject Re: Bug or my lack of understanding? "Reduce output must shrink more rapidly"
Date Wed, 17 Aug 2011 00:37:29 GMT
On Tue, Aug 16, 2011 at 17:03, Chris Stockton <chrisstocktonaz@gmail.com>wrote:

> Hello,
>
> I have been able to reduce a complex case where a certain sized
> document within our application causes "Reduce output must shrink more
> rapidly" errors and I am not sure I understand why. I spent a great
> deal of time making sure I have stripped the database, the documents
> and the views to the bare minimum to make it easy to reproduce, I
> would really appreciate if anyone could give me some insight on what
> is causing this and if a fix exists, may it be ini settings etc. I
> apologize in advanced if this is my lack of understanding views or how
> they work as well as to this email being a bit long, but I think it is
> required to express the issue in case it is indeed a bug.
>
> Kind Regards,
>
> -Chris
>
> --Reproduce steps--
>
> 1) CouchDB Production release 1.10
>
> 2) Create a fresh database
>
> 3) Create the following design document
>
>  {
>   "_id": "_design/test",
>   "_rev": "1-19eb11313c2602a00f0105f78202d1f3",
>   "views": {
>       "Grid": {
>           "map": "function(doc) {\n  emit(\"result\", doc.data);\n}",
>           "reduce": "function(keys, values, rereduce) {\n  var
> container = {};\n\n  if(!rereduce) {\n    for(var value in values) {\n
>     for(var col in values[value]) {\n        if(values[value]) {\n
>      if(!container[col]) {\n            container[col] = {\n
>    total: 0\n            };\n          }\n\n
> container[col].total++;\n        }\n      }\n    }\n  } else {\n
> for(var reduced in values) {\n      for(var col in values[reduced])
> {\n        if(!container[col]) {\n          container[col] = {\n
>     total: 0\n          };\n        }\n\n        container[col].total
> += values[reduced][col].total;\n      }\n    }\n  }\n\n  return
> container;\n}"
>       }
>   },
>   "language": "javascript"
>  }
>
> 4) Create the following regular document (any id is okay)
>  {
>   "_id": "4334dff68f2283e6e8739eabb40a4e7a",
>   "_rev": "24-524e9c9ebeaf88962f41e3a940788610",
>   "data": {
>       "C003089": "c1",
>       "C006990": "c2",
>       "C009996": "c3",
>       "C012132": "c4",
>       "C015574": "c5",
>       "C018908": "c6",
>       "C021545": "c7",
>       "C024392": "c8",
>       "C027281": "c9",
>       "C030392": "c10",
>       "C033457": null,
>       "C036671": null,
>       "C039663": null,
>       "C042967": null,
>       "C045398": null,
>       "C048160": null,
>       "C051924": null,
>       "C054920": null,
>       "C057239": null,
>       "C060993": null,
>       "C063309": null,
>       "C066352": null,
>       "C069003": null,
>       "C072467": null,
>       "C075210": null
>   }
>  }
>
> 5) Call the view, just a typical call no arguments
>  http://<SERVER>:5984/db_24/_design/test/_view/Grid
>
> 6) Verify the response is CORRECT
>
>  {"rows":[{"key":null,"value":{"C003089":{"total":1},"C006990":{"total":1},"C009996":{"total":1},"C012132":{"total":1},"C015574":{"total":1},"C018908":{"total":1},"C021545":{"total":1},"C024392":{"total":1},"C027281":{"total":1},"C030392":{"total":1},"C033457":{"total":1},"C036671":{"total":1},"C039663":{"total":1},"C042967":{"total":1},"C045398":
>
>  {"total":1},"C048160":{"total":1},"C051924":{"total":1},"C054920":{"total":1},"C057239":{"total":1},"C060993":{"total":1},"C063309":{"total":1},"C066352":{"total":1},"C069003":
>   {"total":1},"C072467":{"total":1},"C075210":{"total":1}}}
>  ]}
>
> 7) Now, delete the previous document and add the following:
>  {
>   "_id": "4334dff68f2283e6e8739eabb40a4e7a",
>   "_rev": "24-524e9c9ebeaf88962f41e3a940788610",
>   "data": {
>       "C003089": "c1",
>       "C006990": "c2",
>       "C009996": "c3",
>       "C012132": "c4",
>       "C015574": "c5",
>       "C018908": "c6",
>       "C021545": "c7",
>       "C024392": "c8",
>       "C027281": "c9",
>       "C030392": "c10",
>       "C033457": null,
>       "C036671": null,
>       "C039663": null,
>       "C042967": null,
>       "C045398": null,
>       "C048160": null,
>       "C051924": null,
>       "C054920": null,
>       "C057239": null,
>       "C060993": null,
>       "C063309": null,
>       "C066352": null,
>       "C069003": null,
>       "C072467": null,
>       "C075210": null,
>       "C078387": null
>   }
>  }
>
> 8) Note that all we did was add a single property to the end of
> "data", now run the same view again
>
> 9) Notice the error:
>  {"error":"reduce_overflow_error","reason":"Reduce output must shrink
> more rapidly: Current output:
>
> '[{\"C003089\":{\"total\":1},\"C006990\":{\"total\":1},\"C009996\":{\"total\":1},\"C012132\":{\"total\":1},\"C015574\":'...
> (first 100 of 575 bytes)"}
>
> 10) I am confused because all I did is add a single property, not sure
> how this affects the reduce function?
>

Since you are collecting and creating keys in the output object creating
this single property made the output of reduce larger. CouchDB tries to
detect reduce functions that don't actually reduce the data. If you know for
sure that you are working with a bounded set of properties whose occurrences
you would like to sum you may set reduce_limit=false in your configuration.
The default is true so that users don't shoot themselves in the foot
(especially because you cannot cancel a run-away reduce if you don't have
access to the machine to kill the process).

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message