couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Ghosh <ghosh.debas...@gmail.com>
Subject Re: Query server perfromance issues ..
Date Mon, 21 Sep 2009 12:37:40 GMT
The actual code is something like this ..
var s = isr.readLine
while (s != null) {
    // do stuff
    s = isr.readLine
}

I wrote the other version just to log what I get back. Now this same version
works ok with the earlier version of the couchdb server. That's what beats
me here ..

Thanks.
- Debasish

On Mon, Sep 21, 2009 at 5:46 PM, Robert Newson <robert.newson@gmail.com>wrote:

> I claim you are ignoring null here because of your comment;
>
> while (true) {
>  s = inputstreamreader.readLine
>  if (s == null) // ignore
>  else
>  toJson(s) match {
>   //.. process reset, add_fun etc.
>  }
> }
>
> When System.in is closed this loop will spin; readLine() will always
> return null. Since System.in is only closed when the JVM is exiting,
> it is never correct to ignore it and continue processing.
>
> The loop I presented is not the same as yours as mine will correctly
> exit on process termination.
>
> readLine() *cannot* return null under any circumstance but the close
> of the stream (couchdb cannot pass you null this way). System.in is
> never closed unless the process itself is exiting, and it is never
> reopened.
>
> The mishandling of readLine() is probably hiding the real problem. I
> would guess you pass invalid JSON to couchdb, or fail to return
> anything at all, under some conditions. Couch then kills your view
> server (and would then restart it). The view server, rather than
> gracefully exiting when this happens, will simple spin, never exiting.
>
> B.
>
> On Mon, Sep 21, 2009 at 8:19 AM, Debasish Ghosh
> <ghosh.debasish@gmail.com> wrote:
> > It's in fact referring to a reader that wraps System.in.
> > readLine returns null on end of file, but the earlier version of the
> > snapshot handles it and does not close the query server process. While
> the
> > new server seems to get throttled in the while loop. In fact this is one
> > difference that I forgot to mention. In the earlier version the query
> server
> > does not close, while in the new version it gets closed and restarted for
> > every view operation. Maybe it's getting closed because of the null. I
> can
> > figure that out from the logs. Is this an intentional change in
> > implementation ?
> > Robert -
> > I am not ignoring null. The while loop is very similar to what u mention.
> I
> > switched to the while true version just to log and see if nulls are
> getting
> > returned.
> > Thanks.
> > - Debasish
> >
> > On Mon, Sep 21, 2009 at 3:53 AM, Paul Davis <paul.joseph.davis@gmail.com
> >
> > wrote:
> >>
> >> On Sun, Sep 20, 2009 at 1:34 AM, Debasish Ghosh
> >> <ghosh.debasish@gmail.com> wrote:
> >> > Chris -
> >> > In my query server code, I logged everything that gets exchanged
> between
> >> > the
> >> > couchdb server process and the query server. The difference that I
> >> > noticed
> >> > with the new changes are that the couchdb server sends a huge number
> of
> >> > null
> >> > strings to the view server which chokes the latter. In the snippet
> that
> >> > I
> >> > wrote before ..
> >> >
> >> > while (true) {
> >> >>> >  s = inputstreamreader.readLine  // this reads from stdin
> >> >>> >  if (s == null) // ignore
> >> >>> >  else
> >> >>> >  toJson(s) match {
> >> >>> >    //.. process reset, add_fun etc.
> >> >>> >  }
> >> >>> > }
> >> >
> >>
> >> Does inputstreamreader.readLine refer to this function:
> >>
> >>
> >>
> http://java.sun.com/j2se/1.5.0/docs/api/java/io/BufferedReader.html#readLine%28%29
> >>
> >> If so, and that's returning null, then is it signaling that CouchDB
> >> has tried to close the input stream?
> >>
> >> Paul
> >>
> >> > I put logs in the true branch of if (s == null) and moments later I
> >> > found a
> >> > log created of size 10 MB where the view server gets null strings from
> >> > stdin. This may give some clues towards the problem.
> >> >
> >> > Hope this helps.
> >> > - Debasish
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Sun, Sep 20, 2009 at 10:56 AM, Chris Anderson <jchris@apache.org>
> >> > wrote:
> >> >
> >> >> On Sat, Sep 19, 2009 at 10:09 PM, Debasish Ghosh
> >> >> <ghosh.debasish@gmail.com> wrote:
> >> >> > Yes, actually the reason I brought it up is that the same query
> >> >> > server
> >> >> runs
> >> >> > fine with the earlier version, while it stumbles with the changes
> >> >> > incorporated later. Actually there is a really really big
> difference
> >> >> > in
> >> >> > performance which is primarily because of the timeouts. Thanks
for
> >> >> deciding
> >> >> > to look into it. I will currently stick around with the April
> >> >> > snapshot.Please post your findings on this list - I will be happy
> to
> >> >> upgrade
> >> >> > to the latest.
> >> >> > Thanks.
> >> >> > - Debasish
> >> >>
> >> >> I think what we'll need is a way to get visibility between the beam
> >> >> process and the query server. this could be accomplished with a
> simple
> >> >> log wrapper around the query server, logging both stdin and stdout
to
> >> >> individual files.
> >> >>
> >> >> I like the idea of implementing it as a wrapper because then we can
> >> >> wrap it around the scala as well as the JS query server (and other
> >> >> languages), and get complete transparency into what's going over the
> >> >> wire.
> >> >>
> >> >> This is definitely turning into dev@ territory so I'm moving this
> >> >> thread
> >> >> there.
> >> >>
> >> >> Chris
> >> >>
> >> >> >
> >> >> > On Sun, Sep 20, 2009 at 3:41 AM, Chris Anderson <jchris@apache.org
> >
> >> >> wrote:
> >> >> >
> >> >> >> On Sat, Sep 19, 2009 at 11:40 AM, Debasish Ghosh
> >> >> >> <ghosh.debasish@gmail.com> wrote:
> >> >> >> > Here are some additional behavior changes that I am noticing
> >> >> >> > between
> >> >> the
> >> >> >> 2
> >> >> >> > versions ..
> >> >> >>
> >> >> >> The other big change is in couch_os_process, the addition
of
> >> >> >> couchspawnkillable - maybe this is acting up on your system.
> >> >> >>
> >> >> >> Partially I'm interested in getting to the bottom of this
because
> it
> >> >> >> could be that it's inefficient with the JS query server, but
not
> >> >> >> causing errors, and we just haven't noticed.
> >> >> >>
> >> >> >> > In the newer version, I notice lots of null strings being
sent
> >> >> >> continuously
> >> >> >> > from the couchdb server to the view server. My view server
loop
> >> >> >> > looks
> >> >> >> like
> >> >> >> > the following :-
> >> >> >> >
> >> >> >> > while (true) {
> >> >> >> >  s = inputstreamreader.readLine
> >> >> >> >  toJson(s) match {
> >> >> >> >    //.. process reset, add_fun etc.
> >> >> >> >  }
> >> >> >> > }
> >> >> >> >
> >> >> >> > With the new version, I find lots of null strings coming
in to
> >> >> >> > "s",
> >> >> which
> >> >> >> > makes me include something like the following ..
> >> >> >> >
> >> >> >> > while (true) {
> >> >> >> >  s = inputstreamreader.readLine
> >> >> >> >  if (s == null) // ignore
> >> >> >> >  else
> >> >> >> >  toJson(s) match {
> >> >> >> >    //.. process reset, add_fun etc.
> >> >> >> >  }
> >> >> >> > }
> >> >> >> >
> >> >> >> > And this null business is really huge. Has there been
any change
> >> >> >> > in
> >> >> the
> >> >> >> > protocol between the couchdb server and the view server
? I
> >> >> >> > suspect
> >> >> that
> >> >> >> > these null exchanges are taking up lots of cycles which
result
> in
> >> >> process
> >> >> >> > time out in the new version. I do not get this null stuff
with
> the
> >> >> older
> >> >> >> > version. Is there any chance of such happening with the
changes
> >> >> >> > that
> >> >> have
> >> >> >> > been done in couch_query_servers.erl ?
> >> >> >> >
> >> >> >> > Thanks.
> >> >> >> > - Debasish
> >> >> >> >
> >> >> >> >
> >> >> >> > On Sat, Sep 19, 2009 at 11:34 PM, Debasish Ghosh
> >> >> >> > <ghosh.debasish@gmail.com>wrote:
> >> >> >> >
> >> >> >> >> actually my ["reset"] is not expensive at all ..
it just has a
> >> >> >> array.clear
> >> >> >> >> kind of call.
> >> >> >> >> Just another observation when I run in debug mode
I find that
> >> >> >> >> there
> >> >> are
> >> >> >> >> quite a few cases of OS Process Error {os_process_error,
"OS
> >> >> >> >> process
> >> >> >> timed
> >> >> >> >> out."} being recorded in couch.log. I do not get
this when I am
> >> >> running
> >> >> >> the
> >> >> >> >> earlier version. However no unnatural things appear
in
> >> >> couchdb.stderr.
> >> >> >> My
> >> >> >> >> current setting of os_process_timeout is 20000 ..
I guess
> that's
> >> >> >> >> 20
> >> >> secs
> >> >> >> ..
> >> >> >> >>
> >> >> >> >> Thanks.
> >> >> >> >> - Debasish
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Sat, Sep 19, 2009 at 10:27 PM, Chris Anderson
> >> >> >> >> <jchris@apache.org
> >> >> >> >wrote:
> >> >> >> >>
> >> >> >> >>> On Sat, Sep 19, 2009 at 5:13 AM, Debasish Ghosh
> >> >> >> >>> <ghosh.debasish@gmail.com> wrote:
> >> >> >> >>> > Hi -
> >> >> >> >>> > As I have mentioned previously I have been
working on a
> Scala
> >> >> driver
> >> >> >> for
> >> >> >> >>> > CouchDB, which also includes a Query Server.
I was working
> >> >> >> >>> > with an
> >> >> >> April
> >> >> >> >>> > snapshot of 2009/04/23. This worked fine
for all the views
> and
> >> >> >> >>> validations
> >> >> >> >>> > that I have written.Things were running
fine and I could
> write
> >> >> >> >>> map/reduce
> >> >> >> >>> > and validation functions in Scala.
> >> >> >> >>> > Recently I tried to upgrade to trunk. Suddenly
the views and
> >> >> >> validations
> >> >> >> >>> > became very very slow. After some fact finding,
I tried to
> >> >> >> >>> > poke
> >> >> into
> >> >> >> *
> >> >> >> >>> > couch_query_servers.erl*, since that seemed
to be the
> obvious
> >> >> >> >>> > area
> >> >> to
> >> >> >> >>> look
> >> >> >> >>> > into. I may be worng though, but it was
a blind guess.
> >> >> >> >>> > I noticed that previously I was working
with *revision
> 749852*
> >> >> >> >>> > of
> >> >> the
> >> >> >> >>> file,
> >> >> >> >>> > which delivered the goods for me. Then when
I faced problems
> >> >> >> >>> > with
> >> >> the
> >> >> >> >>> trunk,
> >> >> >> >>> > I started doing a git reset to earlier versions
of this
> file.
> >> >> >> >>> > Now
> >> >> I
> >> >> >> find
> >> >> >> >>> > that it looks like the performance problem
starts from
> >> >> >> >>> > *revision
> >> >> >> 780165*
> >> >> >> >>> of
> >> >> >> >>> > this file. Have a look at
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >>
> >> >>
> >> >>
> http://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_query_servers.erl?r1=780165&r2=749852&diff_format=hfor
> >> >> >> >>> > the difference. Looks like there have been
some major
> changes.
> >> >> >> >>> > I
> >> >> am
> >> >> >> >>> > just
> >> >> >> >>> > wondering if this change has anything to
do with the
> >> >> >> >>> > performance
> >> >> >> issue.
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >> >>> A quick scan of that diff suggests that the only
real behavior
> >> >> change
> >> >> >> >>> that should effect you is the ["reset"] call
for recycled
> >> >> >> >>> processes.
> >> >> >> >>> Maybe reset is expensive in your implementation?
> >> >> >> >>>
> >> >> >> >>> BTW, have you tried running:
> >> >> >> >>>
> >> >> >> >>> spec test/query_server_spec.rb -f specdoc --color
> >> >> >> >>>
> >> >> >> >>> It should be simple to extend that test suite
to test your
> scala
> >> >> >> >>> server. If there are patches we can make to make
it easier to
> >> >> >> >>> integrate outside projects with the query server
test suite,
> I'm
> >> >> happy
> >> >> >> >>> to help there as well.
> >> >> >> >>>
> >> >> >> >>> > Any help, pointer will be appreciated.
> >> >> >> >>> >
> >> >> >> >>> > Thanks.
> >> >> >> >>> > - Debasish
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Chris Anderson
> >> >> >> >>> http://jchrisa.net
> >> >> >> >>> http://couch.io
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Chris Anderson
> >> >> >> http://jchrisa.net
> >> >> >> http://couch.io
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Chris Anderson
> >> >> http://jchrisa.net
> >> >> http://couch.io
> >> >>
> >> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message