httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Fernandez <...@e-group.org>
Subject log split odyssey
Date Fri, 26 Apr 2002 08:10:26 GMT
Hi!

I'm trying to figure out the best way to split a logfile for a statistics system which needs
to take very high numbers of hits per minute - it's for an entire ISP and hosting company
to use everywhere, so it's expecting 2000 hits a second at most peak times. 

The split is based either on the referrer (logs are like logs for an ad server: the referrer
is the site we host, so the stats need to go to that site's owner), or on a site id encoded
in the query string of each request.

I thought of a few ways to do this, but I wonder if I could get some suggestions... 

1) Split with log-rotate every million logfile lines or so, then break it up with a script.
2) Split using the following apache commands:

        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" common
        SetEnvIf Referer spome\.website\.net mydomain-log
        CustomLog /var/log/httpd/statsserver/mydomain_log common env=mydomain-log
        CustomLog /var/log/httpd/pixel-access_log common env=!mydomain-log

all of which are inside the virtual host for my statistics site.
3) Use apache's pipe-log-to-external process feature.
4) Each hit goes to a persistently open cgi in c which writes to it's own logfile - comparing
sids or referrers to what it gets periodically from a file or a database.

Can anyone suggest what is the best/easiest method for achieving this easily, or even a difficult
method, as long as it's proven to work under high loads...

Thanks!

Ale

-- 
Alejandro Fernandez
Electronic Group Interactive
--+34-65-232-8086--

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message