Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 83E3275EE for ; Fri, 5 Aug 2011 08:50:08 +0000 (UTC) Received: (qmail 49364 invoked by uid 500); 5 Aug 2011 08:50:07 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 48552 invoked by uid 500); 5 Aug 2011 08:49:40 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 48527 invoked by uid 99); 5 Aug 2011 08:49:32 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 08:49:32 +0000 Received: from localhost (HELO mail-iy0-f174.google.com) (127.0.0.1) (smtp-auth username rnewson, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 08:49:32 +0000 Received: by iyf40 with SMTP id 40so3176314iyf.5 for ; Fri, 05 Aug 2011 01:49:31 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.47.72 with SMTP id m8mr1501979ibf.164.1312534171487; Fri, 05 Aug 2011 01:49:31 -0700 (PDT) Received: by 10.231.217.13 with HTTP; Fri, 5 Aug 2011 01:49:31 -0700 (PDT) In-Reply-To: References: Date: Fri, 5 Aug 2011 09:49:31 +0100 Message-ID: Subject: Re: unusable database in 1.0.1. steps to take? From: Robert Newson To: user@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Jason, Could you try compressing it with bzip2 at maximum and we'll see if we can move the resulting file around? B. On 4 August 2011 19:24, Jason Konrad wrote: > Couch will start up without any problems. I'm able to access other > databases normally. When requests start trying to interact with the > one problematic db that is when error messages like the one I provided > start to show up in the logs. There was maybe 10 req/sec trying to use > the problem db. After a couple min couch terminates. > > On Thu, Aug 4, 2011 at 9:34 AM, Robert Newson wrote: >> Hm, yeah, that might not be practical. >> >> Does the crash happen immediately on startup or do you need to query a >> document or a view first? >> >> B. >> >> On 4 August 2011 17:07, Jason Konrad wrote: >>> The file system is ext3 and delayed_commits is set to false. >>> >>> I think its possible to share the file although its 65GB so I don't kno= w how >>> practical it would be. >>> On Aug 4, 2011 2:01 AM, "Robert Newson" wrote: >>>> Hi Jason, >>>> >>>> What filesystem is this stored on? Are you running with >>>> delayed_commits set to true or false? >>>> >>>> Finally, are you able to share the database file with the CouchDB >>>> development team? >>>> >>>> B. >>>> >>>> On 4 August 2011 03:39, Jason Konrad wrote: >>>>> I have a database that has become unusable today. The only way I can >>>>> get the couchdb server to function is to remove "thedb" which it >>>>> doesn't like. I've attached a file with some of the log data as well >>>>> as some errorr futon throws up. I'm looking for any ideas for next >>>>> steps to try and "recover" this database. >>>>> >>>>> This is running on a CentOS5 system with erlang-R12B-5.12.el5 and >>>>> couchdb-1.0.1-2.el5 >>>>> >>>>> Let me give you a little context to how this came about. I have one >>>>> database, not the one with problems, that has a high update rate on >>>>> documents. Combined with smallish disks this requires compaction a >>>>> couple times a week. The compactions have been going well but the dat= a >>>>> is not always freed up from the system after the compaction. In order >>>>> to get the disk space to free up I restart couchdb. This process had >>>>> been working for a few months but not today. >>>>> >>>>> On the restart of couchdb the first signs of something out of the >>>>> ordinary was happening, two views were being completely recalulated. >>>>> These calculations can take hours to finish but today that was not th= e >>>>> case. After about 30 min the views stopped calculating and couchdb >>>>> started throwing 500 response codes for everything no matter what >>>>> database was being used ( there are 26 total, 1 bad one ). I tried >>>>> restarting couchdb but all that would happen is the log file would ge= t >>>>> loads of data like the stuff in the attached file and then eventually >>>>> couchdb would terminate. >>>>> >>>>> >>>>> Respect, >>>>> =A0Jason >>>>> >>> >> >