httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dean Gaudet <dgau...@arctic.org>
Subject Re: [PATCH] (take 1) reliable piped logs
Date Wed, 10 Sep 1997 01:04:54 GMT
On Tue, 9 Sep 1997, Dean Gaudet wrote:

> The theory here is that someone will write a nice log processing script
> that does log rotation, vhost splitting, and maybe even parallel DNS
> resolution.  Whatever.  My test case was "|cat >>/tmp/access_log".  The
> point is that this script alone should deal with the vagaries of
> site-specific logging, which keeps Apache's code simple (and lets you
> do things that Apache can't, like asynchronous DNS).

I want to emphasize this more.  piped logs solve at least the following
problems: 

    - thousands of vhosts with log splitting in real-time, without the need
	for thousands of descriptors in apache (thousands of descriptors
	cost you when you fork a CGI and have to close them all, for example).

    - DNS lookups occuring asynchronously with the hit

    - rotation of logs *without stopping the server* -- i.e. the logger can
	start writing somewhere else, the server doesn't know or care.  This
	eliminates any problems with USR1.  In fact, it means the server itself
	only needs to be restarted for configuration changes.

    - filtering of hits which are logged

I think that the ideal logger would:

    - setuid itself to another uid/gid, different than the webserver (it
	starts off as root)

    - setrlimit its descriptors to the max available

    - split logs based on the first word of each line, and maintain a cache
	of currently open logs (so that it's not limited at all by the number
	of descriptors)

    - spawn its own children and use them to resolve DNS on the fly, a la
	squid's DNS children.  (This is a portable solution.)

    - include regex matching of lines to be logged/not-logged

    - die gracefully when given a SIGTERM (i.e. stop reading the log pipe
	and flush all its output buffers, and die FAST)

    - be written in C (perl is fine if you have cpu to waste, but you probably
	don't have cpu to waste)

    - be included with apache-1.3, or be available on apache.org ;)

The only thing that held me back from suggesting this before is that Apache
wasn't able to recover well from a dead logging child.

Notice that such a program feeding off stdin when combined with something
like netcat or djb's tcpclient/tcpserver gives you a method of logging
remotely fairly robustly.

Dean


Mime
View raw message