Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: Very slow queries to view in design doc
From: Sami Sierla <sami.sierla@poplatek.fi>
In-Reply-To: 
 <CABvT1DFG0G02_cUVfHc2=NLKxHJiAeroJK-AC2r_DSp1abCN1w@mail.gmail.com>
Date: Wed, 13 Mar 2013 14:10:55 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <85446807-E2A1-4FD3-9728-9EBB2AC1EDD1@poplatek.fi>
References: <265B0514-CB04-4C5A-8824-77B3008F4D47@poplatek.fi>
 <CABvT1DH=g=MV-BzLMogVLastc=MJtLo6EbLt_mNQsKB=1ygLTw@mail.gmail.com>
 <60C7D1C6-DE78-4798-9E8F-AB03E2F25C7D@poplatek.fi>
 <CABvT1DG6QOyGqFBZD_RyrMvogczgnCxpw1OMsYUNRf7y3FVXNw@mail.gmail.com>
 <5BB0B367-A723-4DC1-8C55-29E14CCC990A@poplatek.fi>
 <CABvT1DFG0G02_cUVfHc2=NLKxHJiAeroJK-AC2r_DSp1abCN1w@mail.gmail.com>
To: user@couchdb.apache.org

Hi,

In the end we just chose to drop reduce function (which was a bit =
overkill) completely, fetch all view entries (there are usually less =
than 100 entries / query)  and do summary calculation on Java side.

-Sami

On Mar 12, 2013, at 6:04 PM, Robert Newson <rnewson@apache.org> wrote:

> Your reduce function does not reduce.
>=20
> I recommend looping in your map function to emit a row for each of the
> attributes you wish to sum and then use the built-in "_sum" reduce
> function. I think you'll find it not only much faster, but it won't
> break.
>=20
> B.
>=20
> On 12 March 2013 04:50, Sami Sierla <sami.sierla@poplatek.fi> wrote:
>> Hi,
>>=20
>> Issue happened again today. I was able to check file descriptors and =
established tcp connections to CouchDB. file descriptors in use were =
about 80 so they are not hitting any limits. Number of connections to =
couch before and after reboot were about 130.
>>=20
>> Reduce function counts sums from document attributes - there are =
about one million documents in view.
>>=20
>> Here's the raw json of map / reduce function:
>>=20
>> "Summary": {
>>         "map": "function(doc) { if (doc.objectType =3D=3D =
\"Document\" && (doc.severity !=3D=3D 4)) { emit(doc.alertTime, =
{\"feeds\": doc.feeds, \"severity\": doc.severity}); } }",
>>         "reduce": "function(keys, values, rereduce) {\n  var i, j, k, =
v, res, feeds;\n  res =3D {};\n  if (!rereduce) {\n    res[\"*,*\"] =3D =
values.length;\n    for (i =3D 0; i < values.length; i++) {\n      k =3D =
\"*,\"+values[i].severity;\n      res[k] =3D (res[k] || 0) + 1;\n      =
feeds =3D values[i].feeds;\n      for (j =3D 0; j < feeds.length; j++) =
{\n        k =3D feeds[j]+\",*\";\n        res[k] =3D (res[k] || 0) + =
1;\n        k =3D feeds[j]+\",\"+values[i].severity;\n        res[k] =3D =
(res[k] || 0) + 1;\n      }\n    }\n  } else {\n     for (i =3D 0; i < =
values.length; i++) {\n        v =3D values[i];\n        for (k in v) =
{\n           res[k] =3D (res[k] || 0) + v[k];\n        }\n     }\n  }\n =
 return res;\n}\n"
>>      }
>>=20
>> View entries are like this:
>> =
{"id":"ac:431590","key":"2010-11-01T00:30:02.000+0000","value":{"feeds":["=
BNK"],"severity":2}}
>>=20
>> Again, this works just fine after reboot (even with reduce) but seems =
to slow over time (about 24-48 hours) until it starts to timeout.
>>=20
>> -Sami
>>=20
>> On Mar 11, 2013, at 11:20 PM, Robert Newson <rnewson@apache.org> =
wrote:
>>=20
>>> Can you access /_log on the server? might have information.
>>>=20
>>> Does the custom reduce function actually reduce (that is, return a
>>> much smaller value than the input 'values' array)?
>>>=20
>>> B.
>>>=20
>>> On 11 March 2013 03:27, Sami Sierla <sami.sierla@poplatek.fi> wrote:
>>>> Hi,
>>>>=20
>>>> There is custom reduce function in this view but result is same =
even if reduce=3Dfalse. Could this have something to do with running out =
of file descriptors (I unfortunately don't have sufficient user rights =
to check this on server). Issue began after we added two databases =
(unrelated to this view) to CouchDB (both have continuous replication to =
standby server).
>>>>=20
>>>> Best,
>>>> Sami
>>>>=20
>>>> On Mar 8, 2013, at 4:20 PM, Robert Newson <rnewson@apache.org> =
wrote:
>>>>=20
>>>>> Do you have a custom reduce function? Are the keys/values in that =
view
>>>>> very large?
>>>>>=20
>>>>> On 8 March 2013 08:18, Sami Sierla <sami.sierla@poplatek.fi> =
wrote:
>>>>>> Hi,
>>>>>>=20
>>>>>> We are having odd issue with CouchDB 1.2.0 view queries - queries =
to one particular view are very slow (tens of seconds with limit=3D1) =
but queries to other views in the same design doc perform as they =
should. Issue goes away if we restart CouchDB but comes back after few =
hours. Views are periodically refreshed so there are no indexing =
blocking the query. The setup we have consists of about ten CouchDB =
databases on active db server (RHEL 5) and second standby server with =
continuous CouchDB pull replication. Same view query works fine on the =
standby server that has no other processes running.
>>>>>>=20
>>>>>> Any ideas what could cause this?
>>>>>>=20
>>>>>> Best,
>>>>>> Sami Sierla
>>>>=20
>>=20