httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Fernandez <>
Subject log split odyssey
Date Fri, 26 Apr 2002 08:10:26 GMT

I'm trying to figure out the best way to split a logfile for a statistics system which needs
to take very high numbers of hits per minute - it's for an entire ISP and hosting company
to use everywhere, so it's expecting 2000 hits a second at most peak times. 

The split is based either on the referrer (logs are like logs for an ad server: the referrer
is the site we host, so the stats need to go to that site's owner), or on a site id encoded
in the query string of each request.

I thought of a few ways to do this, but I wonder if I could get some suggestions... 

1) Split with log-rotate every million logfile lines or so, then break it up with a script.
2) Split using the following apache commands:

        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" common
        SetEnvIf Referer spome\.website\.net mydomain-log
        CustomLog /var/log/httpd/statsserver/mydomain_log common env=mydomain-log
        CustomLog /var/log/httpd/pixel-access_log common env=!mydomain-log

all of which are inside the virtual host for my statistics site.
3) Use apache's pipe-log-to-external process feature.
4) Each hit goes to a persistently open cgi in c which writes to it's own logfile - comparing
sids or referrers to what it gets periodically from a file or a database.

Can anyone suggest what is the best/easiest method for achieving this easily, or even a difficult
method, as long as it's proven to work under high loads...



Alejandro Fernandez
Electronic Group Interactive

The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:> for more info.
To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message