couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Thro <Chris.T...@citrix.com>
Subject RE: stuck indexer process causes view to stop working
Date Thu, 01 Oct 2015 18:01:27 GMT
So it just happened again and in the logs I see the following:
[Thu, 01 Oct 2015 17:55:01 GMT] [info] [<0.191.0>] Starting index update for db: desktopnotification
idx: _design/deskto
pnotification
[Thu, 01 Oct 2015 17:55:01 GMT] [info] [<0.191.0>] Index update finished for db: desktopnotification
idx: _design/deskto
Pnotification

The indexer through shows it started on 2015-10-1 10:55:03 and has made no progress.
17:55:01 GMT is 10:55:01 PST 

We are running these on aws.

-----Original Message-----
From: Chris Thro [mailto:Chris.Thro@citrix.com] 
Sent: Thursday, October 01, 2015 9:42 AM
To: user@couchdb.apache.org
Subject: RE: stuck indexer process causes view to stop working

We are on 1.6.1
I think this is the erland version R14B04 We are unable to produce the problem at will. Can
you clarify what would constitute a minimal environment?

-----Original Message-----
From: Alexander Shorin [mailto:kxepal@gmail.com]
Sent: Thursday, October 01, 2015 8:57 AM
To: user@couchdb.apache.org
Subject: Re: stuck indexer process causes view to stop working

Hi,

I'm not sure, but I think the root of evil is somewhere near here:

[Mon, 28 Sep 2015 12:11:31 GMT] [debug] [<0.86.0>] New task status for
<0.13964.26>: [{changes_done,2},
                                                      {database,

<<"desktopnotification">>},
                                                      {design_document,

<<"_design/desktopnotification">>},
                                                      {progress,200},
                                                      {started_on,1443442291},
                                                      {total_changes,1},
                                                      {type,indexer},
                                                      {updated_on,1443442291}]


Note that progress is 200, while it cannot go beyond 100. So it seems like there is a bug.
At least this looks quite suspicious.

Few more questions:
- What's your CouchDB and Erlang versions?
- Can you try to upgrade to latest CouchDB release 1.6.1 and see if it still happens?
- Can you provide minimal environment to reproduce the issue?

--
,,,^..^,,,


On Thu, Oct 1, 2015 at 6:51 PM, Chris Thro <Chris.Thro@citrix.com> wrote:
> It happened again on the same nodes.  There is still nothing in the logs.  Are there
any bug related to stuck indexer?
>
> -----Original Message-----
> From: Chris Thro [mailto:Chris.Thro@citrix.com]
> Sent: Wednesday, September 30, 2015 3:45 PM
> To: user@couchdb.apache.org
> Subject: RE: stuck indexer process causes view to stop working
>
> Thank you for the follow up.  I have checked the logs and I do see an errors.
> I only see the following at the time the indexer got stuck (but this pops up a lot in
the logs) [Mon, 28 Sep 2015 12:11:31 GMT] [debug] [<0.19987.0>] OS Process #Port<0.8565>
Output :: true [Mon, 28 Sep 2015 12:11:31 GMT] [debug] [<0.86.0>] New task status for
<0.13964.26>: [{changes_done,2},
>                                                       {database,
>                                                        <<"desktopnotification">>},
>                                                       {design_document,
>                                                        <<"_design/desktopnotification">>},
>                                                       {progress,200},
>                                                       {started_on,1443442291},
>                                                       {total_changes,1},
>                                                       {type,indexer},
>                                                       
> {updated_on,1443442291}]
>
>
> One other thing I forgot to mention.  This happened on two separate hosts that were setup
to replicate between each other.
>
> -----Original Message-----
> From: Alexander Shorin [mailto:kxepal@gmail.com]
> Sent: Wednesday, September 30, 2015 2:17 PM
> To: user@couchdb.apache.org
> Subject: Re: stuck indexer process causes view to stop working
>
> Hi,
>
> Have you check the logs for details? If there was an error that prevented indexer correct
work, it have to be there.
>
> --
> ,,,^..^,,,
>
> On Thu, Oct 1, 2015 at 12:14 AM, Chris Thro <Chris.Thro@citrix.com> wrote:
>
>> Hello,
>>
>> I had an issue the other day where if you ran a GET against a view it 
>> would hang indefinitely and if you viewed the view in futon it showed 
>> no rows and put up a dialog box saying error no reponse.  On the 
>> status page there was an indexer running on _*design/<db*_name> that 
>> started at 9/28 5:11am and the last update said 9/28 5:11am and the 
>> progress was 0 out of 9 changes (0%) this was at least 5 hours past the start time
of the process.
>> I ran killall couchjs as there were a lot of couchjs processes, but 
>> they just respawaned and nothing changed.  I eventually had to 
>> restart the couchdb and that finally removed the indexer process from 
>> the status page and allowed the view to work.  Do you have any idea 
>> what might have caused this issue? Has anyone seen something similar before?
>>
>>
>>
>> Thank you
>>
>>
>>
>> *Chris Thro*
>>
>>
>> Senior Database Administrator, Operations
>>
>> *T*: +1 805-690-7925 | *M*: +1 707-364-0682 chris.thro@citrix.com
>>
>>
>>
>> <http://www.citrix.com/>
>>
>>
>>
Mime
View raw message