Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of mikeal.rogers@gmail.com
 designates 209.85.214.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RKWAC1zFmDz1OMHrZ6mbSXm5/rqrbWk6ylb0QFWNo1njNxXu8lnrYQW7IW+rABMDzM
         i7lQeyqeE9IRPhX6bpQS+Odo5wcFwgv8zUSCLwoOmYGOFUo0Zr9+Lp7e6ka+aWv49M3C
         d/0nWG6+WRkmPY2GUw8RD67Op3SWfWeZhYWZ4=
MIME-Version: 1.0
In-Reply-To: <39747FB4-7B59-4D6C-8EA9-C8366DB10AD2@gmail.com>
References: <AANLkTinOG1EcSwObBllXkCaYzt7agSUANCFz52bqt1vt@mail.gmail.com>
	<39747FB4-7B59-4D6C-8EA9-C8366DB10AD2@gmail.com>
Date: Mon, 12 Jul 2010 15:52:21 -0700
Message-ID: <AANLkTim_kyd875S2E2AI6RckSF0775lJJGnXkCh_EA6B@mail.gmail.com>
Subject: Re: CouchDB Crashing under high CPU wait
From: Mikeal Rogers <mikeal.rogers@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=00504501751905b8b7048b389bd4

--00504501751905b8b7048b389bd4
Content-Type: text/plain; charset=ISO-8859-1

We saw this at Mozilla a bunch using CentOS on a vmware.

If you don't install all the vmware tools there are cases when it will just
kill the CouchDB erlang process for some unknown reason. No logs, no
nothing, and it happens even when it's not under load.

-Mikeal

On Mon, Jul 12, 2010 at 2:47 PM, J Chris Anderson <jchris@gmail.com> wrote:

>
> On Jul 10, 2010, at 2:03 AM, [mRg] wrote:
>
> > Hi all,
> >
> > I wonder if any of you can help me with a problem that has been plaguing
> us
> > for months.
> >
> > We have CouchDB running on 3 virtualised (VMWare) RHEL5 servers. Each
> > instance of couchdb replicating to the other. The disks for these virtual
> > machines sit on top of a HP EVA / SAN.
> >
> > We are noticing that sometimes the disk latency on the SAN can get high
> > (when a lot of snapshots/backups are occuring) which will cause high CPU
> > wait (>4.0) on the machines, at these point CouchDB seem to fail. No
> error
> > message in the logs, it just stops without warning.
> >
>
> This is usually a symptom that Erlang is unable to allocate memory. When
> that happens, it just goes away, no logs, no nothing.
>
> It could be the disk wait is cause mochiweb to continue to accept sockets,
> and allocate processes to handle the connections, until at some point there
> is no memory left to allocate.
>
> One option is to configure the max_connections # to be smaller.
>
> But I am just stabbing in the dark here as to the cause of the memory
> over-usage.
>
> And yes it could be the OS killing Couch for other reasons.
>
> There is a heartbeat option which ought to be the robust fix for this (it
> will reboot couch automatically). Someone else on this list will know better
> than I, how to ensure that it runs.
>
> - Chris
>
> > We've turned full debug logging on and there is nothing in any of the
> logs
> > showing any kind of couchdb error but we're seeing all 3 machines fail at
> > the same time which leads us to believe it may be due to the cpu wait /
> disk
> > IO latency issue. We also have Solr running on these boxes and it doesnt
> > seem to be effected by this.
> >
> > I was wondering if anyone has seen this issue before and if there was
> > anything else we can try (other than monitoring to see if the process is
> not
> > running and restart it). Could some other process on RHEL be killing off
> the
> > couch process ?
> >
> > We have tried with 0.10.1 / 0.10.2 and have now upgraded Erlang and are
> > running with 0.11 but still having exactly the same issues.
> >
> > Any help would be appreciated as we go live at the end of the month after
> a
> > year in development and this is the only outstanding issue and it
> > is perplexing to say the least.
> >
> > Regards
> >
> > Stephen
>
>

--00504501751905b8b7048b389bd4--