Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <2E5B7DDB76C6D547B3791C5FCD3A35E308B993E9@SJCPEX01CL02.citrite.net>
References: 
 <2E5B7DDB76C6D547B3791C5FCD3A35E308B9556D@SJCPEX01CL02.citrite.net>
	<CAHdjip+2U77zumqVJzzppmVi+E7Dmd-J6CG_vqsueQAFOiSWuw@mail.gmail.com>
	<2E5B7DDB76C6D547B3791C5FCD3A35E308B95A28@SJCPEX01CL02.citrite.net>
	<2E5B7DDB76C6D547B3791C5FCD3A35E308B97B38@SJCPEX01CL02.citrite.net>
	<CAHdjipK-SOiLen=hcqDBus9zS4OBWG9vY=+ZFoz=cE3dvXYVQw@mail.gmail.com>
	<2E5B7DDB76C6D547B3791C5FCD3A35E308B98FA1@SJCPEX01CL02.citrite.net>
	<2E5B7DDB76C6D547B3791C5FCD3A35E308B993E9@SJCPEX01CL02.citrite.net>
Date: Thu, 1 Oct 2015 22:06:30 +0200
Message-ID: 
 <CA+pq5fqW0KMtY4kw_VRL4G-VmLsPGZthzZNTF+o+_751092dew@mail.gmail.com>
Subject: Re: stuck indexer process causes view to stop working
From: Sebastian Rothbucher <sebastianrothbucher@googlemail.com>
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
Content-Type: multipart/alternative; boundary=001a1143dda039feec05211095a3

--001a1143dda039feec05211095a3
Content-Type: text/plain; charset=UTF-8

Hi Chris,
when you have docs that are significantly different from the others (x-tra
complex / large) maybe it helps 2 leave them out. Another idea: try tee-ing
out what goes back and forth between the couch main process and the indexer
process (might become VERY large, but maybe the very end provides some
insight).
Hope this helps a little - let us know
Best
   Sebastian

On Thu, Oct 1, 2015 at 8:01 PM, Chris Thro <Chris.Thro@citrix.com> wrote:

> So it just happened again and in the logs I see the following:
> [Thu, 01 Oct 2015 17:55:01 GMT] [info] [<0.191.0>] Starting index update
> for db: desktopnotification idx: _design/deskto
> pnotification
> [Thu, 01 Oct 2015 17:55:01 GMT] [info] [<0.191.0>] Index update finished
> for db: desktopnotification idx: _design/deskto
> Pnotification
>
> The indexer through shows it started on 2015-10-1 10:55:03 and has made no
> progress.
> 17:55:01 GMT is 10:55:01 PST
>
> We are running these on aws.
>
> -----Original Message-----
> From: Chris Thro [mailto:Chris.Thro@citrix.com]
> Sent: Thursday, October 01, 2015 9:42 AM
> To: user@couchdb.apache.org
> Subject: RE: stuck indexer process causes view to stop working
>
> We are on 1.6.1
> I think this is the erland version R14B04 We are unable to produce the
> problem at will. Can you clarify what would constitute a minimal
> environment?
>
> -----Original Message-----
> From: Alexander Shorin [mailto:kxepal@gmail.com]
> Sent: Thursday, October 01, 2015 8:57 AM
> To: user@couchdb.apache.org
> Subject: Re: stuck indexer process causes view to stop working
>
> Hi,
>
> I'm not sure, but I think the root of evil is somewhere near here:
>
> [Mon, 28 Sep 2015 12:11:31 GMT] [debug] [<0.86.0>] New task status for
> <0.13964.26>: [{changes_done,2},
>                                                       {database,
>
> <<"desktopnotification">>},
>                                                       {design_document,
>
> <<"_design/desktopnotification">>},
>                                                       {progress,200},
>
> {started_on,1443442291},
>                                                       {total_changes,1},
>                                                       {type,indexer},
>
> {updated_on,1443442291}]
>
>
> Note that progress is 200, while it cannot go beyond 100. So it seems like
> there is a bug. At least this looks quite suspicious.
>
> Few more questions:
> - What's your CouchDB and Erlang versions?
> - Can you try to upgrade to latest CouchDB release 1.6.1 and see if it
> still happens?
> - Can you provide minimal environment to reproduce the issue?
>
> --
> ,,,^..^,,,
>
>
> On Thu, Oct 1, 2015 at 6:51 PM, Chris Thro <Chris.Thro@citrix.com> wrote:
> > It happened again on the same nodes.  There is still nothing in the
> logs.  Are there any bug related to stuck indexer?
> >
> > -----Original Message-----
> > From: Chris Thro [mailto:Chris.Thro@citrix.com]
> > Sent: Wednesday, September 30, 2015 3:45 PM
> > To: user@couchdb.apache.org
> > Subject: RE: stuck indexer process causes view to stop working
> >
> > Thank you for the follow up.  I have checked the logs and I do see an
> errors.
> > I only see the following at the time the indexer got stuck (but this
> pops up a lot in the logs) [Mon, 28 Sep 2015 12:11:31 GMT] [debug]
> [<0.19987.0>] OS Process #Port<0.8565> Output :: true [Mon, 28 Sep 2015
> 12:11:31 GMT] [debug] [<0.86.0>] New task status for <0.13964.26>:
> [{changes_done,2},
> >                                                       {database,
> >
> <<"desktopnotification">>},
> >                                                       {design_document,
> >
> <<"_design/desktopnotification">>},
> >                                                       {progress,200},
> >
>  {started_on,1443442291},
> >                                                       {total_changes,1},
> >                                                       {type,indexer},
> >
> > {updated_on,1443442291}]
> >
> >
> > One other thing I forgot to mention.  This happened on two separate
> hosts that were setup to replicate between each other.
> >
> > -----Original Message-----
> > From: Alexander Shorin [mailto:kxepal@gmail.com]
> > Sent: Wednesday, September 30, 2015 2:17 PM
> > To: user@couchdb.apache.org
> > Subject: Re: stuck indexer process causes view to stop working
> >
> > Hi,
> >
> > Have you check the logs for details? If there was an error that
> prevented indexer correct work, it have to be there.
> >
> > --
> > ,,,^..^,,,
> >
> > On Thu, Oct 1, 2015 at 12:14 AM, Chris Thro <Chris.Thro@citrix.com>
> wrote:
> >
> >> Hello,
> >>
> >> I had an issue the other day where if you ran a GET against a view it
> >> would hang indefinitely and if you viewed the view in futon it showed
> >> no rows and put up a dialog box saying error no reponse.  On the
> >> status page there was an indexer running on _*design/<db*_name> that
> >> started at 9/28 5:11am and the last update said 9/28 5:11am and the
> >> progress was 0 out of 9 changes (0%) this was at least 5 hours past the
> start time of the process.
> >> I ran killall couchjs as there were a lot of couchjs processes, but
> >> they just respawaned and nothing changed.  I eventually had to
> >> restart the couchdb and that finally removed the indexer process from
> >> the status page and allowed the view to work.  Do you have any idea
> >> what might have caused this issue? Has anyone seen something similar
> before?
> >>
> >>
> >>
> >> Thank you
> >>
> >>
> >>
> >> *Chris Thro*
> >>
> >>
> >> Senior Database Administrator, Operations
> >>
> >> *T*: +1 805-690-7925 | *M*: +1 707-364-0682 chris.thro@citrix.com
> >>
> >>
> >>
> >> <http://www.citrix.com/>
> >>
> >>
> >>
>

--001a1143dda039feec05211095a3--