Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of jchris@gmail.com designates
 209.85.212.171 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=ldfOXLeLzaeHSOInDDdCPCuVXmSjpjRt8v6H58RLfuxwtKBZFc7Bwo81WIjVJZu6b9
         JblJmzlIUUCxIose/CRBX6xedi8hvp2mhqrQzOTxheyAOU1uTmDGVzHEk+r/cgzP0Wfw
         K7b1EqXoTN3+c21MHMkALNce9ybu9GDdkPOFw=
MIME-Version: 1.0
Sender: jchris@gmail.com
In-Reply-To: <8b1c89270909191140o745c2b9bva31b867c4a6be5c3@mail.gmail.com>
References: <8b1c89270909190513r1494523ew6a9cb363269c3f04@mail.gmail.com>
	 <e282921e0909190957n9007a6cjb66525355835b543@mail.gmail.com>
	 <8b1c89270909191104g2806357ajac937ef2ba22de37@mail.gmail.com>
	 <8b1c89270909191140o745c2b9bva31b867c4a6be5c3@mail.gmail.com>
Date: Sat, 19 Sep 2009 15:11:34 -0700
Message-ID: <e282921e0909191511j6e772ab2u891d0619e7b3bfab@mail.gmail.com>
Subject: Re: Query server perfromance issues ..
From: Chris Anderson <jchris@apache.org>
To: user@couchdb.apache.org, dghosh@acm.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sat, Sep 19, 2009 at 11:40 AM, Debasish Ghosh
<ghosh.debasish@gmail.com> wrote:
> Here are some additional behavior changes that I am noticing between the =
2
> versions ..

The other big change is in couch_os_process, the addition of
couchspawnkillable - maybe this is acting up on your system.

Partially I'm interested in getting to the bottom of this because it
could be that it's inefficient with the JS query server, but not
causing errors, and we just haven't noticed.

> In the newer version, I notice lots of null strings being sent continuous=
ly
> from the couchdb server to the view server. My view server loop looks lik=
e
> the following :-
>
> while (true) {
> =A0s =3D inputstreamreader.readLine
> =A0toJson(s) match {
> =A0 =A0//.. process reset, add_fun etc.
> =A0}
> }
>
> With the new version, I find lots of null strings coming in to "s", which
> makes me include something like the following ..
>
> while (true) {
> =A0s =3D inputstreamreader.readLine
> =A0if (s =3D=3D null) // ignore
> =A0else
> =A0toJson(s) match {
> =A0 =A0//.. process reset, add_fun etc.
> =A0}
> }
>
> And this null business is really huge. Has there been any change in the
> protocol between the couchdb server and the view server ? I suspect that
> these null exchanges are taking up lots of cycles which result in process
> time out in the new version. I do not get this null stuff with the older
> version. Is there any chance of such happening with the changes that have
> been done in couch_query_servers.erl ?
>
> Thanks.
> - Debasish
>
>
> On Sat, Sep 19, 2009 at 11:34 PM, Debasish Ghosh
> <ghosh.debasish@gmail.com>wrote:
>
>> actually my ["reset"] is not expensive at all .. it just has a array.cle=
ar
>> kind of call.
>> Just another observation when I run in debug mode I find that there are
>> quite a few cases of OS Process Error {os_process_error, "OS process tim=
ed
>> out."} being recorded in couch.log. I do not get this when I am running =
the
>> earlier version. However no unnatural things appear in couchdb.stderr. M=
y
>> current setting of os_process_timeout is 20000 .. I guess that's 20 secs=
 ..
>>
>> Thanks.
>> - Debasish
>>
>>
>> On Sat, Sep 19, 2009 at 10:27 PM, Chris Anderson <jchris@apache.org>wrot=
e:
>>
>>> On Sat, Sep 19, 2009 at 5:13 AM, Debasish Ghosh
>>> <ghosh.debasish@gmail.com> wrote:
>>> > Hi -
>>> > As I have mentioned previously I have been working on a Scala driver =
for
>>> > CouchDB, which also includes a Query Server. I was working with an Ap=
ril
>>> > snapshot of 2009/04/23. This worked fine for all the views and
>>> validations
>>> > that I have written.Things were running fine and I could write
>>> map/reduce
>>> > and validation functions in Scala.
>>> > Recently I tried to upgrade to trunk. Suddenly the views and validati=
ons
>>> > became very very slow. After some fact finding, I tried to poke into =
*
>>> > couch_query_servers.erl*, since that seemed to be the obvious area to
>>> look
>>> > into. I may be worng though, but it was a blind guess.
>>> > I noticed that previously I was working with *revision 749852* of the
>>> file,
>>> > which delivered the goods for me. Then when I faced problems with the
>>> trunk,
>>> > I started doing a git reset to earlier versions of this file. Now I f=
ind
>>> > that it looks like the performance problem starts from *revision 7801=
65*
>>> of
>>> > this file. Have a look at
>>> >
>>> http://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_query_serv=
ers.erl?r1=3D780165&r2=3D749852&diff_format=3Dhfor
>>> > the difference. Looks like there have been some major changes. I am
>>> > just
>>> > wondering if this change has anything to do with the performance issu=
e.
>>> >
>>>
>>> A quick scan of that diff suggests that the only real behavior change
>>> that should effect you is the ["reset"] call for recycled processes.
>>> Maybe reset is expensive in your implementation?
>>>
>>> BTW, have you tried running:
>>>
>>> spec test/query_server_spec.rb -f specdoc --color
>>>
>>> It should be simple to extend that test suite to test your scala
>>> server. If there are patches we can make to make it easier to
>>> integrate outside projects with the query server test suite, I'm happy
>>> to help there as well.
>>>
>>> > Any help, pointer will be appreciated.
>>> >
>>> > Thanks.
>>> > - Debasish
>>> >
>>>
>>>
>>>
>>> --
>>> Chris Anderson
>>> http://jchrisa.net
>>> http://couch.io
>>>
>>
>>
>


--=20
Chris Anderson
http://jchrisa.net
http://couch.io