Return-Path: Delivered-To: new-httpd-archive@hyperreal.org Received: (qmail 29575 invoked by uid 6000); 5 Mar 1998 15:02:21 -0000 Received: (qmail 29377 invoked from network); 5 Mar 1998 15:02:16 -0000 Received: from slarti.muc.de (193.174.4.10) by taz.hyperreal.org with SMTP; 5 Mar 1998 15:02:16 -0000 Received: (qmail 17552 invoked by uid 66); 5 Mar 1998 15:01:56 -0000 Received: by en1.engelschall.com (Sendmail 8.8.8) for new-httpd@apache.org id QAA04400; Thu, 5 Mar 1998 16:00:29 +0100 (MET) Message-Id: <199803051500.QAA04400@en1.engelschall.com> Subject: [PATCH] Network Socket Support for Logfiles To: new-httpd@apache.org (Apache Developer ML) Date: Thu, 5 Mar 1998 16:00:29 +0100 (MET) From: rse@engelschall.com (Ralf S. Engelschall) Organization: Engelschall, Germany. X-Home: http://www.engelschall.com/ X-Mailer: ELM [version 2.4ME+ PL39 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: new-httpd-owner@apache.org Precedence: bulk Reply-To: new-httpd@apache.org Network Socket Support for Logfiles ----------------------------------- Background: In July 1997 Stanley Gambarin posted a short patch for the http_log.c file for hacking in the ability to use the syntax hostname:port for the filename argument of ErrorLog which leads to the open of a socket to this target. This is a very useful variant of writing out log information, especially for ISPs or large website providers who use a cluster of webservers. Here all Apache servers just log their information to a master machine, providing you the ability to monitor all websites at a single point. Problem of the past: Although this variant of logging is very useful for a lot of people, we didn't incorporated it, because the patch was not very complete. The original patch from Stanley directly changes the open_error_log() function in src/main/http_log.c which leads to the fact that network sockets could only be used for the ErrorLog directive. And it didn't documented the new possible variant. Solution: I've started from scratch today and first wrote a open_socket_log() function in addition to the already existing open_piped_log() function. Then I've just _minimally_ patched mod_log_config.c and the open_error_log() function in http_log.c to use this new open_socket_log() function. Finally the corresponding documentation changes are provided to document this variant of writing logfiles. Effect: For instance when you now have a tool like "socket" or "netcat" installed (FreeBSD and Linux hackers usually have out-of-the-box), you just use $ socket -f -l -s 8000 >>/path/to/master/access_log & $ socket -f -l -s 8001 >>/path/to/master/error_log & on the monitoring site `mastersite' and TransferLog @mastersite:8000 ErrorLog @mastersite:8001 on all Apache's on your cluster and then magically all log informations of the whole cluster is collected at the master site. BTW: This is also very useful when using Apache in Reverse Proxy context. Both for writing the Reverse Proxy delegation logfile to the monitoring site via CustomLog @master:8000 "%{%v/%T}t %h -> %{SERVER}e URL: %U" and to collect the logfiles of the various back-end servers as described above. Ralf S. Engelschall rse@engelschall.com www.engelschall.com Index: src/CHANGES =================================================================== RCS file: /e/apache/REPOS/apache-1.3/src/CHANGES,v retrieving revision 1.687 diff -u -r1.687 CHANGES --- CHANGES 1998/03/04 14:28:25 1.687 +++ CHANGES 1998/03/05 14:36:10 @@ -1,5 +1,12 @@ Changes with Apache 1.3b6 + *) Add the ability to write logfile entries to network sockets in addition + to local files and pipes for ErrorLog (core) and TransferLog/CustomLog + (mod_log_config). This is intended to be used either for clusters of + webservers where one wants to store all error and transfer log + information on a single master site for monitoring or just to on-the-fly + transfer away log information. [Ralf S. Engelschall] + *) Fix mod_rewrite for the ugly API case where sections exist but without any RewriteXXXXX directives. Here mod_rewrite is given no chance by the API to initialise its per-server configuration and thus Index: src/include/http_log.h =================================================================== RCS file: /e/apache/REPOS/apache-1.3/src/include/http_log.h,v retrieving revision 1.25 diff -u -r1.25 http_log.h --- http_log.h 1998/01/21 19:17:38 1.25 +++ http_log.h 1998/03/05 13:50:43 @@ -132,4 +132,6 @@ #define piped_log_write_fd(pl) (fileno((pl)->write_f)) #endif +API_EXPORT(int) open_socket_log(char *target); + #endif /* !APACHE_HTTP_LOG_H */ Index: src/main/http_log.c =================================================================== RCS file: /e/apache/REPOS/apache-1.3/src/main/http_log.c,v retrieving revision 1.48 diff -u -r1.48 http_log.c --- http_log.c 1998/01/07 16:46:05 1.48 +++ http_log.c 1998/03/05 14:37:07 @@ -213,6 +213,11 @@ s->error_log = NULL; } #endif + + else if (*s->error_fname == '@' && strchr(s->error_fname, ':') != NULL) { + s->error_log = fdopen(open_socket_log(s->error_fname+1), "w"); + } + else { fname = server_root_relative (p, s->error_fname); if(!(s->error_log = pfopen(p, fname, "a"))) { @@ -638,3 +643,56 @@ pfclose (pl->p, pl->write_f); } #endif + + +API_EXPORT(int) open_socket_log(char *target) +{ + char host[MAX_STRING_LEN]; + char *port; + struct hostent *he; + struct protoent *pe; + struct sockaddr_in sar; + FILE *fp; + int s; + + /* parse host:port pair */ + ap_cpystrn(host, target, sizeof(host)); + if ((port = strchr(host, ':')) == NULL) { + fprintf(stderr, "httpd: Invalid socket target `%s'\n", target); + exit(1); + } + *port++ = '\0'; + + /* create socket address structure */ + memset((char *)&sar, 0, sizeof(sar)); + sar.sin_family = AF_INET; + sar.sin_port = htons(atoi(port)); + if ((sar.sin_addr.s_addr = inet_addr(host)) == INADDR_NONE) { + if ((he = gethostbyname(host)) == NULL) { + perror("gethostbyname"); + fprintf(stderr, "httpd: Couldn't resolve hostname `%s'\n", host); + exit(1); + } + sar.sin_addr.s_addr = *((u_long *)(he->h_addr_list[0])); + } + + /* create the socket and connect to it */ + if ((pe = getprotobyname("tcp")) == NULL) { + perror("getprotobyname"); + fprintf(stderr, "httpd: Couldn't resolve TCP protocol info\n"); + exit(1); + } + if ((s = socket(AF_INET, SOCK_STREAM, pe->p_proto)) < 0) { + perror("socket"); + fprintf(stderr, "httpd: Couldn't create socket\n"); + exit(1); + } + if (connect(s, (struct sockaddr *)&sar, sizeof(sar)) < 0) { + perror("connect"); + fprintf(stderr, "httpd: Couldn't connect to %s\n", target); + exit(1); + } + + return s; +} + Index: src/modules/standard/mod_log_config.c =================================================================== RCS file: /e/apache/REPOS/apache-1.3/src/modules/standard/mod_log_config.c,v retrieving revision 1.47 diff -u -r1.47 mod_log_config.c --- mod_log_config.c 1998/03/03 08:31:28 1.47 +++ mod_log_config.c 1998/03/05 14:21:35 @@ -912,6 +912,9 @@ } cls->log_fd = piped_log_write_fd(pl); } + else if (*cls->fname == '@' && strchr(cls->fname, ':') != NULL) { + cls->log_fd = open_socket_log(cls->fname+1); + } else { char *fname = server_root_relative(p, cls->fname); if ((cls->log_fd = popenf(p, fname, xfer_flags, xfer_mode)) < 0) { Index: htdocs/manual/mod/core.html =================================================================== RCS file: /e/apache/REPOS/apache-1.3/htdocs/manual/mod/core.html,v retrieving revision 1.103 diff -u -r1.103 core.html --- core.html 1998/02/20 06:52:03 1.103 +++ core.html 1998/03/05 14:27:26 @@ -775,6 +775,8 @@ then it is assumed to be relative to the ServerRoot. If the filename begins with a pipe (|) then it is assumed to be a command to spawn to handle the error log. +If the filename begins with an at-sign (@) then it is assumed to be followed +by a hostname:port pair for opening a socket.

Example:

ErrorLog /dev/null
Index: htdocs/manual/mod/mod_log_config.html =================================================================== RCS file: /e/apache/REPOS/apache-1.3/htdocs/manual/mod/mod_log_config.html,v retrieving revision 1.26 diff -u -r1.26 mod_log_config.html --- mod_log_config.html 1998/03/03 08:38:26 1.26 +++ mod_log_config.html 1998/03/05 14:26:24 @@ -365,6 +365,8 @@
A program to receive the agent log information on its standard input. Note the a new program will not be started for a VirtualHost if it inherits the TransferLog from the main server. +
`@' followed by a hostname:port pair +
A socket to receive the log information. Security: if a program is used, then it will be run under the user who started httpd. This will be root if the server