Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of ghosh.debasish@gmail.com
 designates 209.85.132.246 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:reply-to:in-reply-to:references:from:date:message-id
         :subject:to:cc:content-type;
        b=X+V/TbM2gasqGW09DlmqH7SQ2ysuQAfsNdDRQSMsAqgpzhZIV61KMpTRDR74q1aKiz
         MQLIdR5LCRr+HxioR4a9f6TmwN2o4iLqe8PMI1uCATjOHKGTWl1wLtEk7x9F+FFgw4Xg
         AoShLscPpPswnONNQfEeVs1czt8adj2c9Hwn8=
MIME-Version: 1.0
Reply-To: dghosh@acm.org
In-Reply-To: <8b1c89270909191104g2806357ajac937ef2ba22de37@mail.gmail.com>
References: <8b1c89270909190513r1494523ew6a9cb363269c3f04@mail.gmail.com>
	<e282921e0909190957n9007a6cjb66525355835b543@mail.gmail.com>
	<8b1c89270909191104g2806357ajac937ef2ba22de37@mail.gmail.com>
From: Debasish Ghosh <ghosh.debasish@gmail.com>
Date: Sun, 20 Sep 2009 00:10:41 +0530
Message-ID: <8b1c89270909191140o745c2b9bva31b867c4a6be5c3@mail.gmail.com>
Subject: Re: Query server perfromance issues ..
To: Chris Anderson <jchris@apache.org>
Cc: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=0016e68e910a1949d80473f2971d

--0016e68e910a1949d80473f2971d
Content-Type: text/plain; charset=ISO-8859-1

Here are some additional behavior changes that I am noticing between the 2
versions ..
In the newer version, I notice lots of null strings being sent continuously
from the couchdb server to the view server. My view server loop looks like
the following :-

while (true) {
  s = inputstreamreader.readLine
  toJson(s) match {
    //.. process reset, add_fun etc.
  }
}

With the new version, I find lots of null strings coming in to "s", which
makes me include something like the following ..

while (true) {
  s = inputstreamreader.readLine
  if (s == null) // ignore
  else
  toJson(s) match {
    //.. process reset, add_fun etc.
  }
}

And this null business is really huge. Has there been any change in the
protocol between the couchdb server and the view server ? I suspect that
these null exchanges are taking up lots of cycles which result in process
time out in the new version. I do not get this null stuff with the older
version. Is there any chance of such happening with the changes that have
been done in couch_query_servers.erl ?

Thanks.
- Debasish


On Sat, Sep 19, 2009 at 11:34 PM, Debasish Ghosh
<ghosh.debasish@gmail.com>wrote:

> actually my ["reset"] is not expensive at all .. it just has a array.clear
> kind of call.
> Just another observation when I run in debug mode I find that there are
> quite a few cases of OS Process Error {os_process_error, "OS process timed
> out."} being recorded in couch.log. I do not get this when I am running the
> earlier version. However no unnatural things appear in couchdb.stderr. My
> current setting of os_process_timeout is 20000 .. I guess that's 20 secs ..
>
> Thanks.
> - Debasish
>
>
> On Sat, Sep 19, 2009 at 10:27 PM, Chris Anderson <jchris@apache.org>wrote:
>
>> On Sat, Sep 19, 2009 at 5:13 AM, Debasish Ghosh
>> <ghosh.debasish@gmail.com> wrote:
>> > Hi -
>> > As I have mentioned previously I have been working on a Scala driver for
>> > CouchDB, which also includes a Query Server. I was working with an April
>> > snapshot of 2009/04/23. This worked fine for all the views and
>> validations
>> > that I have written.Things were running fine and I could write
>> map/reduce
>> > and validation functions in Scala.
>> > Recently I tried to upgrade to trunk. Suddenly the views and validations
>> > became very very slow. After some fact finding, I tried to poke into *
>> > couch_query_servers.erl*, since that seemed to be the obvious area to
>> look
>> > into. I may be worng though, but it was a blind guess.
>> > I noticed that previously I was working with *revision 749852* of the
>> file,
>> > which delivered the goods for me. Then when I faced problems with the
>> trunk,
>> > I started doing a git reset to earlier versions of this file. Now I find
>> > that it looks like the performance problem starts from *revision 780165*
>> of
>> > this file. Have a look at
>> >
>> http://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_query_servers.erl?r1=780165&r2=749852&diff_format=hfor
>> > the difference. Looks like there have been some major changes. I am
>> > just
>> > wondering if this change has anything to do with the performance issue.
>> >
>>
>> A quick scan of that diff suggests that the only real behavior change
>> that should effect you is the ["reset"] call for recycled processes.
>> Maybe reset is expensive in your implementation?
>>
>> BTW, have you tried running:
>>
>> spec test/query_server_spec.rb -f specdoc --color
>>
>> It should be simple to extend that test suite to test your scala
>> server. If there are patches we can make to make it easier to
>> integrate outside projects with the query server test suite, I'm happy
>> to help there as well.
>>
>> > Any help, pointer will be appreciated.
>> >
>> > Thanks.
>> > - Debasish
>> >
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
>
>

--0016e68e910a1949d80473f2971d--