Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of ghosh.debasish@gmail.com
 designates 209.85.132.251 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:reply-to:in-reply-to:references:from:date:message-id
         :subject:to:cc:content-type;
        b=cw7HpkxyXL3WjQjgnpVHeSubaVjAMQ1PICzWsDNtWLSj87oc746Zl/A0D40KW0X5gg
         58ajiHku2XiUxYcdJtQfkwA+HnyPtaFbsCcUXdFdQ7NTIyvWg8662Yx/Ipr99xAJ/yEW
         n/nmMpW6aPuMPwF4iKUNS72vI6GrB2fbbwebk=
MIME-Version: 1.0
Reply-To: dghosh@acm.org
In-Reply-To: <e282921e0909191511j6e772ab2u891d0619e7b3bfab@mail.gmail.com>
References: <8b1c89270909190513r1494523ew6a9cb363269c3f04@mail.gmail.com>
	<e282921e0909190957n9007a6cjb66525355835b543@mail.gmail.com>
	<8b1c89270909191104g2806357ajac937ef2ba22de37@mail.gmail.com>
	<8b1c89270909191140o745c2b9bva31b867c4a6be5c3@mail.gmail.com>
	<e282921e0909191511j6e772ab2u891d0619e7b3bfab@mail.gmail.com>
From: Debasish Ghosh <ghosh.debasish@gmail.com>
Date: Sun, 20 Sep 2009 10:39:05 +0530
Message-ID: <8b1c89270909192209u58d61e33m8c41de7504da3b1f@mail.gmail.com>
Subject: Re: Query server perfromance issues ..
To: Chris Anderson <jchris@apache.org>
Cc: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=001636b2b48e6dee2f0473fb5e2a

--001636b2b48e6dee2f0473fb5e2a
Content-Type: text/plain; charset=ISO-8859-1

Yes, actually the reason I brought it up is that the same query server runs
fine with the earlier version, while it stumbles with the changes
incorporated later. Actually there is a really really big difference in
performance which is primarily because of the timeouts. Thanks for deciding
to look into it. I will currently stick around with the April
snapshot.Please post your findings on this list - I will be happy to upgrade
to the latest.
Thanks.
- Debasish

On Sun, Sep 20, 2009 at 3:41 AM, Chris Anderson <jchris@apache.org> wrote:

> On Sat, Sep 19, 2009 at 11:40 AM, Debasish Ghosh
> <ghosh.debasish@gmail.com> wrote:
> > Here are some additional behavior changes that I am noticing between the
> 2
> > versions ..
>
> The other big change is in couch_os_process, the addition of
> couchspawnkillable - maybe this is acting up on your system.
>
> Partially I'm interested in getting to the bottom of this because it
> could be that it's inefficient with the JS query server, but not
> causing errors, and we just haven't noticed.
>
> > In the newer version, I notice lots of null strings being sent
> continuously
> > from the couchdb server to the view server. My view server loop looks
> like
> > the following :-
> >
> > while (true) {
> >  s = inputstreamreader.readLine
> >  toJson(s) match {
> >    //.. process reset, add_fun etc.
> >  }
> > }
> >
> > With the new version, I find lots of null strings coming in to "s", which
> > makes me include something like the following ..
> >
> > while (true) {
> >  s = inputstreamreader.readLine
> >  if (s == null) // ignore
> >  else
> >  toJson(s) match {
> >    //.. process reset, add_fun etc.
> >  }
> > }
> >
> > And this null business is really huge. Has there been any change in the
> > protocol between the couchdb server and the view server ? I suspect that
> > these null exchanges are taking up lots of cycles which result in process
> > time out in the new version. I do not get this null stuff with the older
> > version. Is there any chance of such happening with the changes that have
> > been done in couch_query_servers.erl ?
> >
> > Thanks.
> > - Debasish
> >
> >
> > On Sat, Sep 19, 2009 at 11:34 PM, Debasish Ghosh
> > <ghosh.debasish@gmail.com>wrote:
> >
> >> actually my ["reset"] is not expensive at all .. it just has a
> array.clear
> >> kind of call.
> >> Just another observation when I run in debug mode I find that there are
> >> quite a few cases of OS Process Error {os_process_error, "OS process
> timed
> >> out."} being recorded in couch.log. I do not get this when I am running
> the
> >> earlier version. However no unnatural things appear in couchdb.stderr.
> My
> >> current setting of os_process_timeout is 20000 .. I guess that's 20 secs
> ..
> >>
> >> Thanks.
> >> - Debasish
> >>
> >>
> >> On Sat, Sep 19, 2009 at 10:27 PM, Chris Anderson <jchris@apache.org
> >wrote:
> >>
> >>> On Sat, Sep 19, 2009 at 5:13 AM, Debasish Ghosh
> >>> <ghosh.debasish@gmail.com> wrote:
> >>> > Hi -
> >>> > As I have mentioned previously I have been working on a Scala driver
> for
> >>> > CouchDB, which also includes a Query Server. I was working with an
> April
> >>> > snapshot of 2009/04/23. This worked fine for all the views and
> >>> validations
> >>> > that I have written.Things were running fine and I could write
> >>> map/reduce
> >>> > and validation functions in Scala.
> >>> > Recently I tried to upgrade to trunk. Suddenly the views and
> validations
> >>> > became very very slow. After some fact finding, I tried to poke into
> *
> >>> > couch_query_servers.erl*, since that seemed to be the obvious area to
> >>> look
> >>> > into. I may be worng though, but it was a blind guess.
> >>> > I noticed that previously I was working with *revision 749852* of the
> >>> file,
> >>> > which delivered the goods for me. Then when I faced problems with the
> >>> trunk,
> >>> > I started doing a git reset to earlier versions of this file. Now I
> find
> >>> > that it looks like the performance problem starts from *revision
> 780165*
> >>> of
> >>> > this file. Have a look at
> >>> >
> >>>
> http://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_query_servers.erl?r1=780165&r2=749852&diff_format=hfor
> >>> > the difference. Looks like there have been some major changes. I am
> >>> > just
> >>> > wondering if this change has anything to do with the performance
> issue.
> >>> >
> >>>
> >>> A quick scan of that diff suggests that the only real behavior change
> >>> that should effect you is the ["reset"] call for recycled processes.
> >>> Maybe reset is expensive in your implementation?
> >>>
> >>> BTW, have you tried running:
> >>>
> >>> spec test/query_server_spec.rb -f specdoc --color
> >>>
> >>> It should be simple to extend that test suite to test your scala
> >>> server. If there are patches we can make to make it easier to
> >>> integrate outside projects with the query server test suite, I'm happy
> >>> to help there as well.
> >>>
> >>> > Any help, pointer will be appreciated.
> >>> >
> >>> > Thanks.
> >>> > - Debasish
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Chris Anderson
> >>> http://jchrisa.net
> >>> http://couch.io
> >>>
> >>
> >>
> >
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>

--001636b2b48e6dee2f0473fb5e2a--