Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Subject: [PATCH]: mod_logio.c
From: Bojan Smojver <bojan@rexursive.com>
To: Apache Dev List <dev@httpd.apache.org>
Content-Type: multipart/mixed; boundary="=-x+LthlVu+6i5aW7ILsES"
Date: 16 Sep 2002 23:44:36 +1000
Message-Id: <1032183877.1205.10.camel@beast.rexursive.com>
Mime-Version: 1.0


--=-x+LthlVu+6i5aW7ILsES
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

This patch is against the current CVS and it enables logging the number
of input/output bytes per request, which is then useful for producing
actual statistics per virtual host or whatever else.

I have tested the module lightly and it works for me. The trick was to
employ both connection and request based filters and link them together.
I did that through connection notes. I'm not sure if this is the correct
way of doing things, but it does seem to produce accurate numbers in my
tests. I would really appreciate if someone could take a look and verify
if this is something that can be worked with.

The quick configuration is:

-------------------------
SetInputFilter LOGIO
SetOutputFilter LOGIO
-------------------------

And also:

-------------------------
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
%I %O" combinedio
-------------------------

Of course, I'll be documenting the whole thing in the next few days...

Bojan

PS. Thanks everyone, especially Justin, for the hints and help!

--=-x+LthlVu+6i5aW7ILsES
Content-Disposition: attachment; filename=mod_logio.patch
Content-Transfer-Encoding: quoted-printable
Content-Type: text/x-patch; name=mod_logio.patch; charset=ISO-8859-1

diff -u --recursive --new-file httpd-2.0-vanilla/modules/filters/config.m4 =
httpd-2.0/modules/filters/config.m4
--- httpd-2.0-vanilla/modules/filters/config.m4	Mon Sep 16 23:17:54 2002
+++ httpd-2.0/modules/filters/config.m4	Mon Sep 16 23:31:23 2002
@@ -6,6 +6,8 @@
=20
 APACHE_MODULE(include, Server Side Includes, , , yes)
=20
+APACHE_MODULE(logio, Log Input/Output, , , no)
+
 APR_ADDTO(LT_LDFLAGS,-export-dynamic)
=20
 APACHE_MODULE(deflate, Deflate transfer encoding support, , , no, [
diff -u --recursive --new-file httpd-2.0-vanilla/modules/filters/mod_logio.=
c httpd-2.0/modules/filters/mod_logio.c
--- httpd-2.0-vanilla/modules/filters/mod_logio.c	Thu Jan  1 10:00:00 1970
+++ httpd-2.0/modules/filters/mod_logio.c	Mon Sep 16 23:29:05 2002
@@ -0,0 +1,211 @@
+/* =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+ * The Apache Software License, Version 1.1
+ *
+ * Copyright (c) 2000-2002 The Apache Software Foundation.  All rights
+ * reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * 3. The end-user documentation included with the redistribution,
+ *    if any, must include the following acknowledgment:
+ *       "This product includes software developed by the
+ *        Apache Software Foundation (http://www.apache.org/)."
+ *    Alternately, this acknowledgment may appear in the software itself,
+ *    if and wherever such third-party acknowledgments normally appear.
+ *
+ * 4. The names "Apache" and "Apache Software Foundation" must
+ *    not be used to endorse or promote products derived from this
+ *    software without prior written permission. For written
+ *    permission, please contact apache@apache.org.
+ *
+ * 5. Products derived from this software may not be called "Apache",
+ *    nor may "Apache" appear in their name, without prior written
+ *    permission of the Apache Software Foundation.
+ *
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED.  IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR
+ * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+ *
+ * This software consists of voluntary contributions made by many
+ * individuals on behalf of the Apache Software Foundation.  For more
+ * information on the Apache Software Foundation, please see
+ * <http://www.apache.org/>.
+ *
+ */
+
+/*
+ * mod_logio.c: Record input/output traffic in bytes, per request
+ *
+ * Written by Bojan Smojver (bojan@rexursive.com)
+ *
+ */
+
+#include "httpd.h"
+#include "http_config.h"
+#include "http_log.h"
+#include "apr_strings.h"
+#include "apr_general.h"
+#include "util_filter.h"
+#include "apr_buckets.h"
+#include "http_request.h"
+#define APR_WANT_STRFUNC
+#include "apr_want.h"
+
+static const char logioFilterName[] =3D "LOGIO";
+static const char logioConnFilterName[] =3D "LOGIO_CONNECTION";
+module AP_MODULE_DECLARE_DATA logio_module;
+
+typedef struct logio_ctx_t {
+    apr_off_t length;
+} logio_ctx;
+
+static apr_status_t logio_out_filter(ap_filter_t *f, apr_bucket_brigade *b=
b) {
+    request_rec *r =3D f->r;
+    logio_ctx *ctx =3D f->ctx;
+    apr_off_t length;
+
+    if (!ctx) { /* Create context */
+        if (r->main) { /* Don't worry about sub-requests */
+            ap_remove_output_filter(f);
+            return ap_pass_brigade(f->next, bb);
+        }
+
+        ctx =3D f->ctx =3D apr_pcalloc(r->pool, sizeof(*ctx));
+    }
+
+    apr_brigade_length (bb, 0, &length);
+
+    if (length > 0)
+        ctx->length +=3D length;
+
+    apr_table_setn(r->notes, "mod_logio_output_bytes",
+                   apr_off_t_toa(r->pool,ctx->length));
+
+    return ap_pass_brigade(f->next, bb);
+}
+
+static apr_status_t logio_in_conn_filter(ap_filter_t *f,
+                                         apr_bucket_brigade *bb,
+                                         ap_input_mode_t mode,
+                                         apr_read_type_e block,
+                                         apr_off_t readbytes) {
+    conn_rec *c =3D f->c;
+    logio_ctx *ctx =3D f->ctx;
+    apr_off_t length;
+    apr_status_t status;
+    const char *length_n;
+
+    if (!ctx) /* Create context */
+        ctx =3D f->ctx =3D apr_pcalloc(c->pool, sizeof(*ctx));
+
+    length_n =3D apr_table_get(f->c->notes, "mod_logio_input_bytes");
+
+    if (!length_n) /* Request filter unset the note */
+        ctx->length =3D 0;
+
+    status =3D ap_get_brigade(f->next, bb, mode, block, readbytes);
+
+    apr_brigade_length (bb, 0, &length);
+
+    if (length > 0)
+        ctx->length +=3D length;
+
+    apr_table_setn(c->notes, "mod_logio_input_bytes",
+                   apr_off_t_toa(c->pool, ctx->length));
+
+    return status;
+}
+
+
+static apr_status_t logio_in_filter(ap_filter_t *f,
+                                      apr_bucket_brigade *bb,
+                                      ap_input_mode_t mode,
+                                      apr_read_type_e block,
+                                      apr_off_t readbytes) {
+    request_rec *r =3D f->r;
+    logio_ctx *ctx =3D f->ctx;
+    apr_off_t length;
+    apr_status_t status;
+
+    if (!ctx) { /* Create context */
+        const char *length_n;
+
+        if (r->main) { /* Don't worry about sub-requests */
+            ap_remove_input_filter(f);
+            return ap_get_brigade(f->next, bb, mode, block, readbytes);
+        }
+
+        ctx =3D f->ctx =3D apr_pcalloc(r->pool, sizeof(*ctx));
+
+        length_n =3D apr_table_get(f->c->notes, "mod_logio_input_bytes");
+
+        if (length_n) {
+            ctx->length =3D atol(length_n);
+            apr_table_unset (f->c->notes, "mod_logio_input_bytes");
+        }
+    }
+
+    status =3D ap_get_brigade(f->next, bb, mode, block, readbytes);
+
+    apr_brigade_length (bb, 0, &length);
+
+    if (length > 0)
+        ctx->length +=3D length;
+
+    apr_table_setn(r->notes, "mod_logio_input_bytes",
+                   apr_off_t_toa(r->pool, ctx->length));
+
+    return status;
+}
+
+static int logio_pre_conn(conn_rec *c){
+    ap_add_input_filter(logioConnFilterName, NULL, NULL, c);
+
+    return OK;
+}
+
+static void register_hooks(apr_pool_t *p) {
+    ap_hook_pre_connection(logio_pre_conn, NULL, NULL, APR_HOOK_MIDDLE);
+
+    ap_register_input_filter(logioConnFilterName, logio_in_conn_filter, NU=
LL,
+                             AP_FTYPE_CONNECTION);
+
+    ap_register_output_filter(logioFilterName, logio_out_filter, NULL,
+                              AP_FTYPE_TRANSCODE + 1);
+    ap_register_input_filter(logioFilterName, logio_in_filter, NULL,
+                             AP_FTYPE_TRANSCODE + 1);
+}
+
+static const command_rec logio_filter_cmds[] =3D {
+    { NULL }
+};
+
+module AP_MODULE_DECLARE_DATA logio_module =3D {
+    STANDARD20_MODULE_STUFF,
+    NULL,                         /* dir config creater */
+    NULL,                         /* dir merger --- default is to override=
 */
+    NULL,                         /* server config */
+    NULL,                         /* merge server config */
+    logio_filter_cmds,            /* command table */
+    register_hooks                /* register hooks */
+};
diff -u --recursive --new-file httpd-2.0-vanilla/modules/loggers/mod_log_co=
nfig.c httpd-2.0/modules/loggers/mod_log_config.c
--- httpd-2.0-vanilla/modules/loggers/mod_log_config.c	Mon Sep 16 23:17:57 =
2002
+++ httpd-2.0/modules/loggers/mod_log_config.c	Mon Sep 16 23:30:18 2002
@@ -153,9 +153,11 @@
  *         'X' =3D connection aborted before the response completed.
  *         '+' =3D connection may be kept alive after the response is sent=
.
  *         '-' =3D connection will be closed after the response is sent.
-           (This directive was %...c in late versions of Apache 1.3, but
-            this conflicted with the historical ssl %...{var}c syntax.)
-*
+ *         (This directive was %...c in late versions of Apache 1.3, but
+ *         this conflicted with the historical ssl %...{var}c syntax.)
+ * %...I:  bytes received, including request and headers, cannot be zero
+ * %...O:  bytes sent, including headers, cannot be zero
+ *
  * The '...' can be nothing at all (e.g. "%h %u %r %s %b"), or it can
  * indicate conditions for inclusion of the item (which will cause it
  * to be replaced with '-' if the condition is not met).  Note that
@@ -620,6 +622,20 @@
     return "-";
 }
=20
+static const char *log_bytes_recv_all(request_rec *r, char *a)
+{
+    const char *length =3D apr_table_get(r->notes, "mod_logio_input_bytes"=
);
+
+    return length ? length : "-";
+}
+
+static const char *log_bytes_sent_all(request_rec *r, char *a)
+{
+    const char *length =3D apr_table_get(r->notes, "mod_logio_output_bytes=
");
+
+    return length ? length : "-";
+}
+
 /*****************************************************************
  *
  * Parsing the log format string
@@ -1367,6 +1383,8 @@
         log_pfn_register(p, "T", log_request_duration, 1);
         log_pfn_register(p, "U", log_request_uri, 1);
         log_pfn_register(p, "s", log_status, 1);
+        log_pfn_register(p, "I", log_bytes_recv_all, 0);
+        log_pfn_register(p, "O", log_bytes_sent_all, 0);
     }
=20
     return OK;

--=-x+LthlVu+6i5aW7ILsES--