httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randy Terbush <>
Subject Re: namelookups and databases
Date Wed, 26 Jul 1995 02:58:06 GMT
> On Tue, 25 Jul 1995, Randy Terbush wrote:
> > I've been attempting to shove my log data into Postgres and am
> > coming to a sobering realization. It has taken 9 hours to process
> > 15,000 requests.....  
> That's a bit sad.  I'm currently working on a web stats package that
> stores the data in an mSQL database (including number of hits/bytes per
> page per date, number of hits from each domain per page per date etc). 
> >From this the stats front end can work through the historical data and
> create whatever report is required.  
> Shoving the data into the database isn't a big problem.  I just parse CLF 
> logs keeping track of the date/time of the last logged access and go from 
> there.  Works pretty well  (eg  -HUP the server at 00:01am and ram 
> yesterday's log down the loading programme).  This is still 
> work-in-progress but it is looking like a good bit of code.

David, after my experience with Postgres, I am planning to bring
up mSQL give it a go on the same task. Knowing you are working on
the same thing, perhaps we should pull together...

Two things I have liked about Postgres that I did not find
in mSQL are no need for table keys, and unlimited (or large)
row lengths. I know you planned to implement the later.

BTW - I am logging a nonstandard format file. I have changed
the date to be an integer+microseconds which could easily be
used as the key. I have also changed the log to be TAB separated
to be more easily digestible for the parsing program.


View raw message