Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
Date: Tue, 4 Oct 2005 13:55:38 +0100
From: Colm MacCarthaigh <colm@stdlib.net>
To: dev@httpd.apache.org
Subject: Re: [PATCH] Re: Pluggable mod_log_config
Message-ID: <20051004125538.GA21579@stdlib.net>
Reply-To: colm@stdlib.net
References: <434150EB.2080007@turner.com> <20051003222814.GA7331@stdlib.net>
 <43427774.5050605@turner.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-15
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <43427774.5050605@turner.com>
User-Agent: Mutt/1.5.9i

On Tue, Oct 04, 2005 at 08:37:08AM -0400, Brian Akins wrote:
> In this case, from my patches:
> 
> LogFormat "INSERT INTO foo VALUES ('%h', '%l');" foo-sql
> 
> CustomLog mysql://user:password@host/database foo-sql
> 
> and the mysql module would get the arrays of strings and lengths.  at 
> init time, it would have prepared the format sql.  At log time, it would 
> bind and execute.

No, I don't think it's so simple at all. Although it would have to parse
the SQL at init time (how does LogFormat know to ask the MySQL provider
to do this?) that can't be simply untied from the actions at log-time;
in the case of SQL for example you have to be very pedantic about the
escaping, so that a request for "/foo\'; nasty sql;" doesn't kill us. 

And we still need additional per-provider directives to provide the
database, hostname, username and so on information. If such directives
are required anyway, what effort are we saving? Why not just replace
mod_log_config rather than plug into it. 

> >And after all of this, what if any, are the compelling reasons to
> >implement this in httpd at all? Why can't all of this be moved into
> >piped loggers? 
> 
> Try pipe loggers with 60 or so virtual hosts.  It doesn't scale well, as 
> we open a pipe for each virtual that defines custom log.

Ahh, that's a different problem; it's trivial to have a piped logger do
the splitting (it's what we do). 

Though if all of this text parsing is getting expensive, I wonder would
anyone be interested in a protocol for binary logging from httpd?

> CustomLog /logs/site.sock common

You re-implemented syslog :-) I've done the same myself for our hosting
service, we use syslog-ng to do all sorts of weird things with the info. 

> define "damn busy." 

We frequently see over 4000 requests per second being logged. Almost any
time there's a major security update for fedora really. 

> We may have a different scale of "busy."  I'v found pipe logs with
> lots of virtuals to be less than spectacular.  In Apache logging will
> be the best performance, but it lacks some flexibility.  That's why I
> wrote my domain socket stuff.  However, I think the patches I've
> submitted allow for in-Apache logging to be very flexible with no
> additional overhead of pipes.

Pipes themselves have little overhead, it's basically shared memory with
a standard IO interface and automatic mutexing. The processes on the
other side certainly need not be cumbersome - and I really like that you
can run a pipe-logger as a different uid, in a chroot and so on, it's a
nice place to sandbox all of that icky SQL parsing and that kind of
thing.

-- 
Colm MacC�rthaigh                        Public Key: colm+pgp@stdlib.net