From user-return-32444-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Fri Mar 26 13:17:26 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 6CDD2180642 for ; Fri, 26 Mar 2021 14:17:26 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 4665745035 for ; Fri, 26 Mar 2021 13:17:25 +0000 (UTC) Received: (qmail 6464 invoked by uid 500); 26 Mar 2021 13:17:23 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 6452 invoked by uid 99); 26 Mar 2021 13:17:22 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Mar 2021 13:17:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 087C8C02D6 for ; Fri, 26 Mar 2021 13:17:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, NUMERIC_HTTP_ADDR=0.001, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id Nl4dhnYXYpQ4 for ; Fri, 26 Mar 2021 13:17:21 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::d2a; helo=mail-io1-xd2a.google.com; envelope-from=fuzzyillogic@gmail.com; receiver= Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id A24A47FCA4 for ; Fri, 26 Mar 2021 13:17:20 +0000 (UTC) Received: by mail-io1-xd2a.google.com with SMTP id b10so5343680iot.4 for ; Fri, 26 Mar 2021 06:17:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=EC0hRz3wXFpydMV9GAlYPw6axNXERYxHcqRdBURhX58=; b=e8daWvadXgo2o9Qq+2X0q3+myi5QBoxpAgAl9qj8ZLbB7CozinPACU6q+g+9M3Nz/H CHW3HJksjnP7fOUBxkmphIte7AUWE7k2txD8p2pg7YEkvIayy3h0R10XwrdrdeVQCMWY ueanw6fzQwtRr23fVCCniXHGbsocWdjZg/XHIq2yix4dVp4LJfQ6oe5N3xdS2wosZdaE vdrK9kgTedWOghzDZ3Q9klT20j3e0MTb+WjrjoAgZ5zPU69//aaQ0PDuHjXV/Tf2suOM CR7th+Wz6EeulSzozqROa52zA48l44AY1YiGjQvX7QCSIfL85ZWpXnlz26fILwnEhp62 BIvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=EC0hRz3wXFpydMV9GAlYPw6axNXERYxHcqRdBURhX58=; b=tFqMY34j4igiRXn2EKPQoBmslZHFnacsXeKa3rBC+mMKvkA0TS+kxKZy5WYLCAaLAb ar7as76i0j8tWPR39cpDu9H+N229VaDRlpoUSof5evpjs+3VMI6zAdDar4Pvmcct1i4g gYJ97+aKNr8XHF2UdpcIhp3vnwK82so0xpgRElLzwwSxCojeN5Pm+YkgL6tSJtlhGhJ6 MYXZR4Tl6nQoHHRj1iD+jUvID7K3bv+LjzyHDrlS+CDXU8Hsb3hGIErR69z8ECHZoOsV ow9t/3akTiIe8v0TVFTpD5pVMw7pSqfESadXylYHXmA+LUkzauipEFYAOBskMxUT7MvJ vyOQ== X-Gm-Message-State: AOAM533FsmK75o4okq11hga0l1MHJ0GKYpbA+RZ54AbDzJzSN4DMtQ/M riAmGq5KvLXfcRsPJZyF3/0sh+ZBG6kNcowlQ69iNR/KTjS0ebLC X-Google-Smtp-Source: ABdhPJzhREjrWOXmRAGZUUEigkrPC1eCwOECEdRiWTlzzOjBroJtqWOfR7G3bX05GvJSA2H5QEge5m8c1UUgmpx+lLo= X-Received: by 2002:a5d:9d56:: with SMTP id k22mr10336553iok.145.1616764639091; Fri, 26 Mar 2021 06:17:19 -0700 (PDT) MIME-Version: 1.0 From: f z Date: Fri, 26 Mar 2021 14:17:08 +0100 Message-ID: Subject: problem with disappearing db To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary="000000000000ff9fa005be705925" --000000000000ff9fa005be705925 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I just had a strange problem with an installation of couchdb. After restarting the couchdb service, it seems it can't access anymore to some of the dbs that were there before. The files of those dbs seems to be still present in the shards folder, but after the restart, it seems the system can't access them. Just before the restart, the dbs were there and working regularly, there was no delete operation. The couchdb configuration, at least to my knowledge, did not change since installation, and by my check seems to be the standard one. Couchdb is version 2.3.1. I've also tried on a different machine copying one of our previous timed-backups (we do a daily snapshot of the whole couch db folders), but once restored seems to have the exact same problem: the couchdb service comes up, but the "documents" and "main" dbs are missing. I've no idea were to start debugging this, and if there is a way of recovering those dbs from the backup I've right now. Anyway, the logs during the restart are the following: [info] 2021-03-26T09:06:57.909241Z couchdb@127.0.0.1 <0.56.0> -------- alarm_handler: {set,{system_memory_high_watermark,[]}} [error] 2021-03-26T09:10:26.491449Z couchdb@127.0.0.1 <0.64.0> -------- gen_server memsup terminated with reason: {port_died,normal} last msg: {'EXIT',<0.65.0>,{port_died,normal}} state: [{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",7946051584},{"Total",8171896832}]}},{items,{"Worst Memory User",[{"Pid",<0.218.0>},{"Memory",7687816}]}}] extra: [] [info] 2021-03-26T09:10:32.894224Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_log started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:32.899154Z couchdb@127.0.0.1 <0.9.0> -------- Application folsom started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.012320Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_stats started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.012544Z couchdb@127.0.0.1 <0.9.0> -------- Application khash started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.026786Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_event started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.027019Z couchdb@127.0.0.1 <0.9.0> -------- Application hyper started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.038401Z couchdb@127.0.0.1 <0.9.0> -------- Application ibrowse started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.048224Z couchdb@127.0.0.1 <0.9.0> -------- Application ioq started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.048468Z couchdb@127.0.0.1 <0.9.0> -------- Application mochiweb started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.064719Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB 2.3.1 is starting. [info] 2021-03-26T09:10:33.064980Z couchdb@127.0.0.1 <0.213.0> -------- Starting couch_sup [notice] 2021-03-26T09:10:33.082657Z couchdb@127.0.0.1 <0.96.0> -------- config: [features] pluggable-storage-engines set to true for reason nil [notice] 2021-03-26T09:10:33.097461Z couchdb@127.0.0.1 <0.96.0> -------- config: [admins] admin set to XXX for reason nil [info] 2021-03-26T09:10:33.258955Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB has started. Time to relax. [info] 2021-03-26T09:10:33.259081Z couchdb@127.0.0.1 <0.212.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/ [info] 2021-03-26T09:10:33.259276Z couchdb@127.0.0.1 <0.9.0> -------- Application couch started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.259725Z couchdb@127.0.0.1 <0.9.0> -------- Application ets_lru started on node 'couchdb@127.0.0.1' [notice] 2021-03-26T09:10:33.290376Z couchdb@127.0.0.1 <0.272.0> -------- rexi_server : started servers [notice] 2021-03-26T09:10:33.295464Z couchdb@127.0.0.1 <0.279.0> -------- rexi_buffer : started servers [info] 2021-03-26T09:10:33.295971Z couchdb@127.0.0.1 <0.9.0> -------- Application rexi started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.342338Z couchdb@127.0.0.1 <0.9.0> -------- Application mem3 started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.342560Z couchdb@127.0.0.1 <0.9.0> -------- Application fabric started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.403837Z couchdb@127.0.0.1 <0.9.0> -------- Application chttpd started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.436538Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_index started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.437227Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_mrview started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.437531Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_plugins started on node 'couchdb@127.0.0.1' [notice] 2021-03-26T09:10:33.559220Z couchdb@127.0.0.1 <0.96.0> -------- config: [features] scheduler set to true for reason nil [info] 2021-03-26T09:10:33.620740Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_replicator started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.649361Z couchdb@127.0.0.1 <0.9.0> -------- Application couch_peruser started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.703726Z couchdb@127.0.0.1 <0.9.0> -------- Application ddoc_cache started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.734466Z couchdb@127.0.0.1 <0.9.0> -------- Application global_changes started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.734529Z couchdb@127.0.0.1 <0.9.0> -------- Application jiffy started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.755574Z couchdb@127.0.0.1 <0.9.0> -------- Application mango started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.777359Z couchdb@127.0.0.1 <0.9.0> -------- Application setup started on node 'couchdb@127.0.0.1' [info] 2021-03-26T09:10:33.777609Z couchdb@127.0.0.1 <0.9.0> -------- Application snappy started on node 'couchdb@127.0.0.1' [notice] 2021-03-26T09:10:38.469290Z couchdb@127.0.0.1 <0.353.0> -------- couch_replicator_clustering : cluster stable [notice] 2021-03-26T09:10:38.478642Z couchdb@127.0.0.1 <0.405.0> -------- Started replicator db changes listener <0.511.0> [error] 2021-03-26T09:10:45.348964Z couchdb@127.0.0.1 emulator -------- Error in process <0.670.0> on node 'couchdb@127.0.0.1' with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"documents",[{fi= le,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1= ,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"s= rc/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fab= ric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1= ,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2021-03-26T09:10:45.349035Z couchdb@127.0.0.1 <0.260.0> -------- Could not get design docs for <<"shards/20000000-3fffffff/documents.1594915751">> error:{database_does_not_exist,[{mem3_shards,load_shards_from_db,"documents= ",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_= disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{f= ile,"src/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"s= rc/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun= -0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [notice] 2021-03-26T09:10:48.291445Z couchdb@127.0.0.1 <0.279.0> -------- rexi_buffer : cluster stable [notice] 2021-03-26T09:10:48.291610Z couchdb@127.0.0.1 <0.272.0> -------- rexi_server : cluster stable [error] 2021-03-26T09:11:06.416502Z couchdb@127.0.0.1 <0.260.0> -------- Could not get design docs for <<"shards/20000000-3fffffff/main.1594915751">> error:{database_does_not_exist,[{mem3_shards,load_shards_from_db,"main",[{f= ile,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,= 1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"= src/mem3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fa= bric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',= 1,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2021-03-26T09:11:06.416806Z couchdb@127.0.0.1 emulator -------- Error in process <0.1116.0> on node 'couchdb@127.0.0.1' with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"main",[{file,"s= rc/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{fi= le,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/me= m3_shards.erl"},{line,54}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_v= iew_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{fi= le,"src/couch_db.erl"},{line,627}]}]} --000000000000ff9fa005be705925--