From user-return-31639-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Fri Aug 17 08:51:24 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F3DA8180627 for ; Fri, 17 Aug 2018 08:51:23 +0200 (CEST) Received: (qmail 79412 invoked by uid 500); 17 Aug 2018 06:51:22 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 79400 invoked by uid 99); 17 Aug 2018 06:51:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Aug 2018 06:51:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 218DF1A18B0 for ; Fri, 17 Aug 2018 06:51:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.271 X-Spam-Level: X-Spam-Status: No, score=0.271 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.972] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=messagingengine.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id xv3auSvLqFdM for ; Fri, 17 Aug 2018 06:51:20 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A81925F3B4 for ; Fri, 17 Aug 2018 06:51:20 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 765FE2102F; Fri, 17 Aug 2018 02:51:20 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 17 Aug 2018 02:51:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=IspRS9 fc7crDUMe9YMhNVFd2sW/9jtmOynCr4EaNHGc=; b=LAhMcDkwyQJ5+jnHZn7UQc 4bD4O2CxJEcRqv+6cdOXlDpQ0eS4q1q0WXZWPhwrr+wEGY41ZDDdjmE0LPXecv6j FGbQ6u4Pvo9KgvDvQnIW5k9HZ/YCCaj2Vg4vcrbyCOZf1s9E510ffAn7S4BviQOK VTpSfrM8+oc9szv+vvfQrx+scYueXmsVQzilOf6WCwJ5nMN9WxSUzs9mOpuN4bn1 U3+Y15tnIPWFfFRI8y21I+7k/4HQl4QawdmEOB53qBu+t7pDRx/R7YvJbPNLULbs tkDldmA443f/2Re5xzxvs/PqWRA3ItTDMgHYzT5tKH5h1ObEKCLsXHfmwBwTOEig == X-ME-Proxy: X-ME-Sender: Received: from [192.168.0.23] (cpc87379-slou4-2-0-cust152.17-4.cable.virginm.net [82.4.22.153]) by mail.messagingengine.com (Postfix) with ESMTPA id 5E20D10269; Fri, 17 Aug 2018 02:51:19 -0400 (EDT) From: Robert Samuel Newson Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Couch 2.x cluster returning inconsistent _all_docs Date: Fri, 17 Aug 2018 07:51:17 +0100 References: <941872DC-5AF0-408D-A10F-0558E2AF2DC5@apache.org> <1937615737.389.1534467188003.JavaMail.Joan@BRAIN> To: user , Joan Touzet In-Reply-To: <1937615737.389.1534467188003.JavaMail.Joan@BRAIN> Message-Id: <5584C3E1-7CA0-4EEF-9798-556F219F3926@apache.org> X-Mailer: Apple Mail (2.3445.9.1) You're quite right, Joan.=20 B. > On 17 Aug 2018, at 01:53, Joan Touzet wrote: >=20 > Hey everyone, >=20 > Doesn't 'emfile' mean too many open file handles? Arif, check your = file handle limit as well as permissions on the files, see: >=20 > = http://docs.couchdb.org/en/stable/maintenance/performance.html#maximum-ope= n-file-descriptors-ulimit >=20 > Finally, we have a very good bit of documentation now that improves on = Robert's excellent SO post, we recommend using these instructions now = instead:=20 >=20 > http://docs.couchdb.org/en/stable/cluster/sharding.html >=20 > -Joan "yay good documentation" Touzet >=20 > ----- Original Message ----- > From: "Robert Samuel Newson" > To: "user" > Sent: Thursday, August 16, 2018 5:03:51 PM > Subject: Re: Couch 2.x cluster returning inconsistent _all_docs >=20 > the word 'emfile' indicates the immediate problem is one of file = permissions. The user that couchdb is running as is unable to open the = shards/5... file. So you probably need a recursive chmod/chown session = to fix up ownership and permissions. >=20 > Secondly, you have changed the names of 2 nodes. This is ... unwise. = All clustered databases address their data files using the node names, = so what you've effectively done is delete 2 of the 3 copies of your = databases, which would explain the weird inconsistencies. >=20 > I wrote a stackoverflow post a while ago on how to correctly move an = individual shard which explains some of the internals: = https://stackoverflow.com/questions/6676972/moving-a-shard-from-one-bigcou= ch-server-to-another-for-balancing. >=20 > For your situation, I believe you will need to update all the = documents in the _dbs database and substitute your old node names for = the new node names. I strongly advise you take a backup of everything = you can. >=20 > For others observing this thread, I strongly advise against renaming = nodes like this, it can only lead to trouble, and potentially data loss. >=20 > B. >=20 >> On 16 Aug 2018, at 19:25, Arif Khan wrote: >>=20 >> emfile >=20