couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arif Khan <ar...@atvenu.com>
Subject Re: Couch 2.x cluster returning inconsistent _all_docs
Date Fri, 17 Aug 2018 16:59:50 GMT
Hey,

Thanks everyone for the inputs. Silly as it is, we fixed the issue by
restarting one of our couch instances in the cluster

@Robert: re: "For your situation, I believe you will need to update all the
documents in the _dbs database and substitute your old node names for the
new node names. I strongly advise you take a backup of everything you can"
- I did update all the documents in the _dbs database  (thats what I meant
by updating cluster metadata) when I added the new nodes using a script.
They all have the documents right always but were disagreeing only on the
_all_docs view. I am not sure why that happened

I will work with the error logs next. Attached is the ruby script I used to
update cluster metadata to move all shards from one node to another.

Regards
Arif

On Fri, Aug 17, 2018 at 12:51 AM, Robert Samuel Newson <rnewson@apache.org>
wrote:

> You're quite right, Joan.
>
> B.
>
> > On 17 Aug 2018, at 01:53, Joan Touzet <wohali@apache.org> wrote:
> >
> > Hey everyone,
> >
> > Doesn't 'emfile' mean too many open file handles? Arif, check your file
> handle limit as well as permissions on the files, see:
> >
> >    http://docs.couchdb.org/en/stable/maintenance/
> performance.html#maximum-open-file-descriptors-ulimit
> >
> > Finally, we have a very good bit of documentation now that improves on
> Robert's excellent SO post, we recommend using these instructions now
> instead:
> >
> >    http://docs.couchdb.org/en/stable/cluster/sharding.html
> >
> > -Joan "yay good documentation" Touzet
> >
> > ----- Original Message -----
> > From: "Robert Samuel Newson" <rnewson@apache.org>
> > To: "user" <user@couchdb.apache.org>
> > Sent: Thursday, August 16, 2018 5:03:51 PM
> > Subject: Re: Couch 2.x cluster returning inconsistent _all_docs
> >
> > the word 'emfile' indicates the immediate problem is one of file
> permissions. The user that couchdb is running as is unable to open the
> shards/5... file. So you probably need a recursive chmod/chown session to
> fix up ownership and permissions.
> >
> > Secondly, you have changed the names of 2 nodes. This is ... unwise. All
> clustered databases address their data files using the node names, so what
> you've effectively done is delete 2 of the 3 copies of your databases,
> which would explain the weird inconsistencies.
> >
> > I wrote a stackoverflow post a while ago on how to correctly move an
> individual shard which explains some of the internals:
> https://stackoverflow.com/questions/6676972/moving-a-
> shard-from-one-bigcouch-server-to-another-for-balancing.
> >
> > For your situation, I believe you will need to update all the documents
> in the _dbs database and substitute your old node names for the new node
> names. I strongly advise you take a backup of everything you can.
> >
> > For others observing this thread, I strongly advise against renaming
> nodes like this, it can only lead to trouble, and potentially data loss.
> >
> > B.
> >
> >> On 16 Aug 2018, at 19:25, Arif Khan <arifk@atvenu.com> wrote:
> >>
> >> emfile
> >
>
>

Mime
View raw message