httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Morin <...@ilabsinc.com>
Subject RE: [users@httpd] Apache 2.x configuration for high load servers
Date Wed, 25 May 2011 19:07:47 GMT
Ya sorry about the attachments... I put URLs to my personal website rather
than the work related one to be safe :)

http://www.stupidguytalk.org/WEB3.ireel.com-apache_processes-day.png

http://www.stupidguytalk.org/WEB3.ireel.com-apache_accesses-day.png

Sorry about that....


Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207




-----Original Message-----
From: Rob Morin [mailto:rob@ilabsinc.com] 
Sent: Wednesday, May 25, 2011 3:00 PM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache 2.x configuration for high load servers

Thanks for the prompt reply Mark, I will try to add a few things and clear
up some stuff I might have omitted in my first post...

I have server-status enabled as well as Munin for Apache..

I have attached a munin graphs to this email, not sure if its proper or
not... sorry if not... one is accesses and one is processes per day..
To be safe its about 120 accesses per second , each server is basically the
same give or take 5%, I will enable access logging for one domain and see
what happens with respect to the time to serve a request...

Here is a paste of server status for the same server as the attached pngs of
munin.. our busy time will start in the next couple hours....

Current Time: Wednesday, 25-May-2011 13:48:42 EDT Restart Time: Sunday,
22-May-2011 04:02:05 EDT Parent Server Generation: 1 Server uptime: 3 days 9
hours 46 minutes 37 seconds Total accesses: 24212224 - Total Traffic: 7.1 GB
CPU Usage: u1544.11 s494.57 cu0 cs0 - .692% CPU load
82.2 requests/sec - 25.5 kB/second - 316 B/request
46 requests currently being processed, 384 idle workers

_C____._..____._..__C___..__C___.___.____._.._.__._._._..____.__
__C_._..____._.__.____________.___._______________.___._.__.R.__
._C____CC___C_.__CCR.._._..WW_____.._.C_..C..__.._.__.._______._
__._..__._.__.____.C__C._____..__.__C.____C___._______.C______._
.C_._._.__.________C._____.._..__._..__.C_C__.._.___.__.._____C_
_____._._C__.________.._____.__R_.____.._C_.C.__.CCC.______.__._
._C_____....__.___..__.____.__.__.____._____...____._._._.__.__C
._C______..______._._._________._.___..____....CC_.__C_C_.C__._.
R__...C.__._.___..._.___C_______....__._____._____._____..._.._C
__W.__.__.________..___.


So then my current config I have seems good enough then, from what you have
spoken about so far? 

StartServers       150
MinSpareServers    125
MaxSpareServers   250
ServerLimit      600
MaxClients       600
MaxRequestsPerChild   1500

As the load at busy times is about 9, and this is for quad core, so there
might be room for improvement somewhere I guess.. This is why I turned off
logging as we have about 60 domains all writing logfiles via rotatelogs I
fogured removing that extra file open and writing to would help a bit... I
also made a small RAMdisk to store the eAccelerator cache files rather than
use disk... we also use memcache to store sessions and for mysql querries..
We also use Edgecast as our CDN for static content, like images and CSS
files..

We also use New Relic and new PHP performance tuning  tool, its very cool...
that shows that a good 95% of our requests take under 200 ms while the rest
can vary between 600 ms and way up to 10 seconds, the 10 second ones are
where we connect to processors to valaidate information on clients..

I will read over that in the URL your pasted and see what I can do to still
increase performance...

As for the logs just to clarify how I did it, all the virtual host config
files have the CustomLog entries commented out, with this in minf that means
Apache will send access log detaisl to whatever the defautl access log file
is , its normally something like /var/log/hhpd/access_log, so in the
httpd.con file under CustomLog combined , I just have %h, then in the
location for customlog rather than /var/log/httpd/access_log I have
/dev/null so this way it will og minimum info and then not write it to disk
and send it out to the bit bucket.... we do nto need logs as we use Google
Analytics in our code. Is this ok, as if I just remove or comment out the
access_log loacation or have no combined entry apache will nto restart....
if I remove mod_log then I will get no errors logs, and those are more
important than access_logs for us...

Hope this sheds more light on my problem...

Thanks again for all your help...

Rob Morin
Systems Administrator
Infinity Labs Inc.
(514) 387-0638 Ext: 207




-----Original Message-----
From: Mark Montague [mailto:mark@catseye.org]
Sent: Wednesday, May 25, 2011 12:06 PM
To: users@httpd.apache.org
Cc: Rob Morin
Subject: Re: [users@httpd] Apache 2.x configuration for high load servers

  On May 25, 2011 10:27 , Rob Morin <rob@ilabsinc.com> wrote:
> We have recently had a 30% increase in traffic and will be expecting 
> more. I would like to know the correct way to calculate the proper 
> settings for prefork.c my settings, here is what I have now...

> I am going to add more RAM today to make a total of 24 gigs on each 
> server, and i am going to add an additional web server in the next 
> week or so...
>
> Each day we get about 5 million visits...
>

The settings you use for the prefork MPM are not determined by how many
visits you get per day, but by how many concurrent requests the server in
question needs to handle during your busiest periods. Number of concurrent
requests is in turn determined by how quickly requests come in together with
how long it takes to serve (process, respond to,
handle) each request.

For example, let's say that 90% of the requests you serve require 1 second
to serve, but 10% of the requests take 5 seconds to serve. Let's also assume
that during your busiest period you are getting 100 requests per second.
In each second, then, 100 children will be needed to server these requests
-- 90 of these children will finish (and become available to serve another
request) after 1 second and the remainder will become available again after
5 seconds. This means that after 5 seconds of your busiest anticipated load,
140 children will be tied up serving requests, and hence you'd want to set
ServerLimit comfortably above this value (to allow margin for error and
future growth), assuming that you have the necessary hardware resources
(RAM, processor cores) to effectively do so. In addition, you may want to
set MinSpareServers high enough to handle a spike in load equivalent to one
second's worth of requests, and set MaxSpareServers to free up resources if
more than the number of children required to server two seconds worth of
requests are sitting idle.

Access logs with a custom log format containing %D (time required to serve
the request in microseconds) are your best tool for figuring out the request
patterns that your web server needs to be able to handle. 
Also, mod_status is a useful tool for seeing what each child process is
doing at single point in time (which children are serving requests, which
children are idle, and so on).


> Apache is currently using 6207 Megs of memory...
>
> There are 245 Apache processes running at this time.
>
> --------------------------------
>
> So the load is high, but i have free RAM left and only running 245 
> Apache processes, out of a possible 600? I only show the first few 
> lines of top ..
>

This is anticipated. Apache HTTP Server's prefork MPM will try to keep the
number spare processes (children that are idle and waiting for new
requests) between MinSpareServers and MaxSpareServers at all times.

The rule of thumb I recommend is that you want your system load to be less
than twice the number of processor cores. Other people's advice may differ
from this, so I recommend that you ask around.

If your system load is more than twice the number of processor cores, I
recommend:

- Determine if there is any storage (disk, etc.) I/O bottleneck; if so,
eliminate it.
- Determine if there is any network I/O bottleneck (network link, load
balancer, firewall, etc.); if so, eliminate it.
- Reduce the time required to serve each requests:
-- Reduce time required by any web applications run by the web server
(active content)
-- Reduce time required by the web server itself (web server configuration,
operating system configuration)

The above recommendations may result in you adding more I/O capacity, more
RAM, more processor cores, and/or more machines to your web server cluster.
However, I believe it is generally better to only add more resources once
you have determined that all existing resources are being used as optimally
as possible: improve any web applictions or other active content on your web
site so that they require less resources; change your infrastructure
architecture (what each web server does and how it does it including caching
and proxying); and improve the configuration or your web server (and other
servers, including any database servers) and your operating system.


> Or do I get this wrong, does each connection have an httpd process 
> that is run, or are child/subsequent connections, not listed on under 
> a ps ax??
>

If you are using the prefork MPM, a number of child processes get created to
handle web server requests (connections). The number of these will vary
dynamically based on how you configure the prefork MPM and also on how many
requests come in how quickly. All of the processes will show up under ps.


> I removed access logging, I disable access logging by commenting out 
> the CustomLog directive in each virtualhost and then in httpd.com I 
> set CustomLog value to /dev/null, this way I can keep error working 
> too. is this ok? Here aer some of the httpd.con entries.
>

Are you sure that you want to not log web server accesses? This removes an
incredibly important diagnostic tool. Also note that if you use CustomLog
with a file of /dev/null, then you are having Apache HTTP Server actually
create each log entry and send it to the kernel, which will then discard it
instead of writing it to disk; if you really want to turn off access
logging, then it would be better to completely remove any CustomLog or
TransferLog directives from your web server configuration.

If you want to improve performance, see the recommendations at
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html


> Here are loaded modules, I am not sure if I need them all, can I 
> safely turn any off?

You can and should turn off any modules you are not using; this will 
save some memory and may make your web server a little more secure. You 
will need to look at what functionality your web site currently relies 
upon, together with your web server configuration, in order to determine 
which modules you can avoid loading.

--
   Mark Montague
   mark@catseye.org


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message