httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dirk-Willem van Gulik <di...@webweaving.org>
Subject RE: Log rotation/Logging to a database?
Date Wed, 21 Aug 2002 09:23:39 GMT

In general, and as a compromise between reliability, performance,
robustness and functionality, you are propably best of logging 'simply' to
disk and using a graceful restart.

Given that you are only talking about 150 live children; that is not a
very high rate/large number. Plus a mature OS, such as *BSD or Solaris can
handle in the thousands with no ill effects. So doubling or even
increasing it by a ten fold with a piped log application is no issue at
all. Generally; get worried if you are hitting the 5 to 10's of thousands
of processes.

You could increase the size/switch on BUFFERED_LOGS to reduce the # of
writes; bear in mind that a typical SCSI disk is hard pressed to do more
than 2-3000 writes/second. I.e. if each hit is a write that becomes a
distinct cap.

Would you be committing to a database at similar rates - that may be an
issue. So ultimately you want to look at the peak number of hits coming
out of your server. Other options are to send the data to one or more
separate processes (along the line of a pipe to rotatelog) where for
example the data is buffered or pre-processed to be reduced in size. I.e.
certain things are already counted and the raw data is dropped.

Certain database types/deamons/protocols are desigined for very high rates
- you may want to look into those; but once you hit those you may find
that you have dedicated 100Mbit switching fabric/connections between your
web farm and your logging/auditing backend; and having the latter equiped
with fancy scsi/raid systems.

At that point you are propably saturating several T1's and doing in the
several million of hits/day (A million hits per day is 'only' 11.2 hits
per second!) or you are having a 'weird' load distribution during the day.

Dw

On Wed, 21 Aug 2002, Koen Vingerhoets wrote:

> why don't you use a customlog to limit the size?
>
> 	SetEnvIf Request_URI \.*$ dontlog
> 	CustomLog /home/logs/access_log common env=!dontlog
>
> -----Original Message-----
> From: Jon Benson [mailto:Jon@destra.com]
> Sent: 21 August 2002 10:26
> To: 'users@httpd.apache.org'
> Subject: Log rotation/Logging to a database?
>
>
> Hi folks,
>
> Here is my dilemma:
>
> I've recently started in this role to find I'm responsible for 2 apache
> servers, each with hundreds (eg 1200 on one) of virtual sites and NO log
> rotation occuring!  :(
>
> Hence log files are growing out of control and consuming valuable resources
> (disc space).
>
> Due to the time it would take to stop apache, rotate all the logs, and start
> apache again that's not an option.
>
> Due to the number of simultaneous requests the server/s may endure (I've
> seen 150+ simultaneous instances of httpd) using rotatelogs or cronolog does
> not seem to be an option.  Ie due to the extra two processes created and
> destroyed per httpd instance.  Though perhaps I'm over estimating the
> overhead this would create and need to do some load testing?
>
> I attempted to create a little script to use cronosplit and essentially
> rotate the logs in place but stress testing it showed, as expected, there
> would be corrupt lines and lines out of order if I ran it while a httpd
> process was attempting to log to the same file.
>
>
> What I'm currently considering:
> pgLOGd - http://www.digitalstratum.com/pglogd/index.php
>
> I'm just beginning to wonder if the overhead of a Postgres database is going
> to be greater then the overhead of an extra 300 processes and the associated
> open file handles?
>
> Ie pgLOGd would be used and log files generated from the database at regular
> intervals (eg daily)
>
> Any/all comments, particularly from those who may have encountered a similar
> dilemma or had experience with pgLOGd, would be most welcome.
>
>
> Thanks,
>
> Jon Benson
> Mail/DNS/Linux Administrator
> OzHosting.com
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message