Return-Path: 
 <users-return-88402-apmail-httpd-users-archive=httpd.apache.org@httpd.apache.org>
Delivered-To: apmail-httpd-users-archive@www.apache.org
Received: (qmail 9064 invoked from network); 20 May 2009 09:06:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 May 2009 09:06:38 -0000
Received: (qmail 70644 invoked by uid 500); 20 May 2009 09:06:47 -0000
Delivered-To: apmail-httpd-users-archive@httpd.apache.org
Received: (qmail 70616 invoked by uid 500); 20 May 2009 09:06:47 -0000
Mailing-List: contact users-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@httpd.apache.org
list-help: <mailto:users-help@httpd.apache.org>
list-unsubscribe: <mailto:users-unsubscribe@httpd.apache.org>
List-Post: <mailto:users@httpd.apache.org>
List-Id: <users.httpd.apache.org>
Delivered-To: mailing list users@httpd.apache.org
Received: (qmail 70605 invoked by uid 99); 20 May 2009 09:06:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 May 2009 09:06:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of icicimov@gmail.com designates
 209.85.221.127 as permitted sender)
Received: from [209.85.221.127] (HELO mail-qy0-f127.google.com)
 (209.85.221.127)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 May 2009 09:06:37 +0000
Received: by qyk33 with SMTP id 33so481123qyk.10
        for <users@httpd.apache.org>; Wed, 20 May 2009 02:06:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=rf0t0zuGKwAekvKeoTJQTtQF3Ge/Hok3Bi085XQu0u8=;
        b=b9exzT7WsMnTUiI42LJFaEV5+CsQ6RzjJQ/t34vUYvFnh5/uGni7o6mEbTZ12dR7sH
         weeucI09CzPTy7BslAIGcf61Z1uvifLBIwZivI6iklyQM0ZH1gUzITeeX+QbNOQF9Y2H
         HCisH61VUfK0l1QUAtFID87kEnxvyh+XdRsV4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nIh7KdjGfK0liKGhXuEoU6aHsrIz60FIxWX9KxJWZccdmTc7qZsSz9x3kWuKxalIHz
         O5V9iK6tqOlfleShG2dHKQ9tk6qnpys+YnsDmjJNAK+3XOqeSS+8uDFXJ505g0ek+doK
         +Lfi2pSm6SbO0rhsFXn2bNPE4xHOgPkGVLdAQ=
MIME-Version: 1.0
Received: by 10.220.92.205 with SMTP id s13mr1227354vcm.31.1242810376583;
 Wed,
	20 May 2009 02:06:16 -0700 (PDT)
In-Reply-To: <a5ad29010905192300m68301ee7s878cf6927b11d1f6@mail.gmail.com>
References: <a5ad29010905192300m68301ee7s878cf6927b11d1f6@mail.gmail.com>
Date: Wed, 20 May 2009 19:06:16 +1000
Message-ID: <f89deef00905200206t3c7782d6g849ca4ce940665cf@mail.gmail.com>
From: Igor Cicimov <icicimov@gmail.com>
To: users@httpd.apache.org
Content-Type: multipart/alternative; boundary=00163631080504fac7046a5457d7
X-Virus-Checked: Checked by ClamAV on apache.org
Subject: Re: [users@httpd] What is the best way to handle "too many open
	files" errors?

--00163631080504fac7046a5457d7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Have you considered increasing the open files limit? Have a look at the
ulimit command. The default is sometimes not enough.

Cheers,

Igor

On Wed, May 20, 2009 at 4:00 PM, Ben Welsh <ben.welsh@gmail.com> wrote:

> Hello,
>
> I'm managing a pgsql --> django --> mod_python --> apache machine that
> serves no media, but handles the postgres memcached and apache part of the
> stack all by itself. It's a dedicated virtual server with Red Hat EL and 1GB
> RAM.
>
> As traffic to the site has ramped up over time, I've begun to encounter "too
> many open files in the system" errors via Apache that look something like
> this:
>
> OperationalError: FATAL: could not open file <filename>: Too many open
> files in systemThis has become something of a debilitating problem and I'm
> struggling to come up with the best strategy for handling it. Some people
> have told me that I can solve it by better configuring my existing machine
> -- while others have insisted that I must upgrade my server rig by expanding
> it beyond one DV.
>
> My hopes in posting here is that y'all might give me some insight into how
> I can answer the question about the better approach.
>
> I've run the commands you see below and the conclusion that I'm drawing is
> that the I simply have way too many httpd file objects open -- but I'm
> unsure about the wisest way of reducing them -- or whether I can reasonably
> expected them to go down much.
>
> I'd very much appreciate any insight the list could provide into the
> situation, even if it's just point me to a reference. I've trolled the lists
> and the web and still am a Iittle unsure of the best approach, so I thought
> I'd post. If I'm overlooking the most awesome tutorial or previous lister
> post ever, please forgive my ignorance. I'd appreciate even just the link.
>
> Thank you,
>
> Ben.
>
> $ ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> file size               (blocks, -f) unlimited
>  pending signals                 (-i) 1024
> max locked memory       (kbytes, -l) 32
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 137215
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> *postgres*.conf (max_files_per_process is commented out)
> max_connections = 100
> shared_buffers = 1000
>
> $ httpd -l
>
> Compiled in modules:
>   core.c
>   prefork.c
>   http_core.c
>   mod_so.c
>
> httpd.conf...
>
> Timeout 20
> KeepAlive Off
> MaxKeepAliveRequests 100
> KeepAliveTimeout 5
> <IfModule prefork.c>
> StartServers       8
> MinSpareServers    5
> MaxSpareServers   20
> ServerLimit      256
> MaxClients       256
> MaxRequestsPerChild  4000
> </IfModule>
>
> $ cat /proc/sys/fs/file-nr
> 73320   0       262144
>
> $ lsof | awk '{print $1}' | sort | uniq -c
>       1 COMMAND
>      10 awk
>      49 bash
>      17 crond
>    5154 httpd
>       9 init
>      25 lsof
>      39 memcached
>     807 postmaste
>      62 saslauthd
>      69 sendmail
>      28 sh
>       7 sort
>     183 sshd
>      13 syslogd
>       8 uniq
>      19 xinetd
>
>

--00163631080504fac7046a5457d7
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Have you considered increasing the open files limit? Have a look at the uli=
mit command. The default is sometimes not enough.<br><br>Cheers,<br><br>Igo=
r<br><br><div class=3D"gmail_quote">On Wed, May 20, 2009 at 4:00 PM, Ben We=
lsh <span dir=3D"ltr">&lt;<a href=3D"mailto:ben.welsh@gmail.com">ben.welsh@=
gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello,<br><br><sp=
an>I&#39;m managing a pgsql --&gt; django --&gt; mod_python --&gt; apache m=
achine that serves no media, but handles the postgres memcached and apache =
part of the stack all by itself. It&#39;s a dedicated virtual server with R=
ed Hat EL and </span>1GB RAM. <br>

<span><br>As traffic to the site has ramped up over time, I&#39;ve begun to=
 encounter </span><span>&quot;too many open files in the system&quot; error=
s</span> via Apache that look something like this:<br><br><h1 style=3D"marg=
in-top: 0.25em;">

<font size=3D"2">OperationalError: FATAL: could not open file &lt;filename&=
gt;: Too many  open files in system</font></h1>This has become something of=
 a debilitating problem and I&#39;m struggling to come up with the best str=
ategy for handling it. Some people have told me that I can solve it by bett=
er configuring my existing machine -- while others have insisted that I mus=
t upgrade my server rig by expanding it beyond one DV. <br>

<br>My hopes in posting here is that y&#39;all might give me some insight i=
nto how I can answer the question about the better approach. <br><br>I&#39;=
ve run the commands you see below and the conclusion that I&#39;m drawing i=
s that the I simply have way too many httpd file objects open -- but I&#39;=
m unsure about the wisest way of reducing them -- or whether I can reasonab=
ly expected them to go down much. <br>

<br> I&#39;d very much appreciate any insight the list could provide into t=
he situation, even if it&#39;s just point me to a reference. I&#39;ve troll=
ed the lists and the web and still am a Iittle unsure of the best approach,=
 so I thought I&#39;d post. If I&#39;m overlooking the most awesome
tutorial or previous lister post ever, please forgive my ignorance. I&#39;d
appreciate even just the link.<br><br>Thank you,<br><br>Ben.<br><br><span>$=
 ulimit -a</span><br><span>core file size=A0=A0=A0=A0=A0=A0=A0=A0=A0 (block=
s, -c) 0</span><br><span>data seg size=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (kbyte=
s, -d) unlimited</span><br><span>file size=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0 (blocks, -f) unlimited</span><br>

<span>
pending signals=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (-i) 1024</=
span><br><span>max locked memory=A0=A0=A0=A0=A0=A0 (kbytes, -l) 32</span><b=
r><span>max memory size=A0=A0=A0=A0=A0=A0=A0=A0 (kbytes, -m) unlimited</spa=
n><br><span>open files=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0 (-n) 1024</span><br>

<span>pipe size=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (512 bytes, -p) 8</span><b=
r><span>
POSIX message queues=A0=A0=A0=A0 (bytes, -q) 819200</span><br><span>stack s=
ize=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (kbytes, -s) 10240</span><br><sp=
an>cpu time=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (seconds, -t) unlimit=
ed</span><br><span>max user processes=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0 (-u) 137215</span><br>

<span>virtual memory=A0=A0=A0=A0=A0=A0=A0=A0=A0 (kbytes, -v) unlimited</spa=
n><br><span>
file locks=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (=
-x) unlimited<br><br></span><span><b>postgres</b>.conf (max_files_per_proce=
ss is commented out)</span><br><span>max_connections =3D 100</span><br><spa=
n>shared_buffers =3D 1000</span><br>
<br><span>$ httpd -l
</span><br><p><span> Compiled in modules:
</span><br><span> =A0 core.c
</span><br><span> =A0 prefork.c
</span><br><span> =A0 http_core.c
</span><br><span> =A0 mod_so.c
</span><br><span> </span></p><p><span>httpd.conf...
</span><br><span> </span></p><p><span>Timeout 20
</span><br><span> KeepAlive Off
</span><br><span> MaxKeepAliveRequests 100
</span><br><span> KeepAliveTimeout 5
</span><br><span> </span></p><span>&lt;IfModule prefork.c&gt;
</span><br><span> StartServers =A0 =A0 =A0 8
</span><br><span> MinSpareServers =A0 =A05
</span><br><span> MaxSpareServers =A0 20
</span><br><span> ServerLimit =A0 =A0 =A0256
</span><br><span> MaxClients =A0 =A0 =A0 256
</span><br><span> MaxRequestsPerChild =A04000
</span><br><span> &lt;/IfModule&gt;
</span><br><br><span>$ cat /proc/sys/fs/file-nr</span><br><span>73320=A0=A0=
 0=A0=A0=A0=A0=A0=A0 262144</span><br><br><span>$ lsof | awk &#39;{print $1=
}&#39; | sort | uniq -c</span><br><span>=A0=A0=A0=A0=A0 1 COMMAND</span><br=
><span>=A0=A0=A0=A0 10 awk</span><br>

<span>=A0=A0=A0=A0 49 bash</span><br><span>=A0=A0=A0=A0 17 crond</span><br>=
<span>=A0=A0 5154 httpd</span><br><span>=A0=A0=A0=A0=A0 9 init</span><br><s=
pan>
=A0=A0=A0=A0 25 lsof</span><br><span>=A0=A0=A0=A0 39 memcached</span><br><s=
pan>=A0=A0=A0 807 postmaste</span><br><span>=A0=A0=A0=A0 62 saslauthd</span=
><br><span>=A0=A0=A0=A0 69 sendmail</span><br><span>=A0=A0=A0=A0 28 sh</spa=
n><br><span>=A0=A0=A0=A0=A0 7 sort</span><br><span>=A0=A0=A0 183 sshd</span=
><br>

<span>=A0=A0=A0=A0 13 syslogd</span><br><span>=A0=A0=A0=A0=A0 8 uniq</span>=
<br><span>=A0=A0=A0=A0 19 xinetd</span><br><br>
</blockquote></div><br>

--00163631080504fac7046a5457d7--