httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Darroch <>
Subject rotatelogs input from named pipe
Date Tue, 18 Dec 2012 22:22:16 GMT
Hi --

   I recently needed to wire up httpd's rotatelogs to a named pipe (a FIFO)
instead of to stdin.  For context, we have a non-httpd server process
configured to write to the pipe instead of a log file; further, this
server likes to intermittently open, write to, and close its various
log files.

   I wanted/needed to open the pipe in non-blocking mode, so rotatelogs
would start right away (we're using rotatelogs -cf) instead of blocking
until a writer process happened to open the pipe to write some log data.

   Also, because the server process repeatedly opens and closes its
log files/pipes, a normal reader will exit on EOF after the first close,
and therefore miss all subsequent writes.

   I played around for some time with variants on "tail -f pipe | rotatelogs"
and "cat pipe | rotatelogs" but both have problems.  The tail -f seems
to miss data, especially at first; I can't explain that but it's what
I observed.  The cat command works to capture all data while running,
but it blocks at startup and (worse for us) exits on the first EOF.

   What I really wanted was a simpler command line, passing the pipe to
rotatelogs directly, and having it (a) open in non-blocking mode so
the -cf options are effective right away, and (b) ignore EOFs until I kill
it explicitly.  The latter is ungainly but that's how the server also
has to be stopped, via kill -TERM, so it's acceptable in this case for us.

   First I puzzled over how to open a pipe in non-blocking mode from the
start using APR, and ended up with a set of patches for that; see for the details,
but the gist is an apr_file_namedpipe_open() function to do this.
Probably not sufficiently full of trademark portable goodness, but
effective for the moment.

   Next the following patch adds a "-i inputpipe [-o]" set of options,
when possible, so that rotatelogs will listen to the pipe instead of
stdin and will ignore EOFs.

   I'm posting both the APR and httpd patches because I thought it was
an interesting issue, and I suspect there are alternate and perhaps
better approaches; certainly the APR patch isn't ideal.  But after
a long time spent digging into the source code for tail and cat and
trying many variants, this now seems to work smoothly for us, at least.


--- support/rotatelogs.c.orig	2012-12-18 13:44:02.000000000 -0800
+++ support/rotatelogs.c	2012-12-18 14:11:56.000000000 -0800
@@ -67,6 +67,8 @@
     int truncate;
     const char *linkfile;
     const char *postrotate_prog;
+    const char *inputpipe;
+    int ignore_eof;
     int create_empty;
@@ -101,9 +103,9 @@
-            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t] [-e] [-c] <logfile>
+            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i inputpipe [-o]] [-f] [-t] [-e]
[-c] <logfile> "
-            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t] [-e] <logfile> "
+            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i inputpipe] [-f] [-t] [-e] <logfile>
             "{<rotation time in seconds>|<rotation size>(B|K|M|G)} "
             "[offset minutes from UTC]\n\n",
@@ -137,7 +139,9 @@
             "  -f       Force opening of log on program start.\n"
             "  -t       Truncate logfile instead of rotating, tail friendly.\n"
             "  -e       Echo log to stdout for further processing.\n"
+            "  -i path  Use specified named pipe as input.\n"
+            "  -o       Hold input pipe open when writers exit (with -i only).\n"
             "  -c       Create log even if it is empty.\n"
@@ -194,7 +198,9 @@
     fprintf(stderr, "Rotation verbose:            %12s\n", config->verbose ? "yes" : "no");
     fprintf(stderr, "Rotation create empty logs:  %12s\n", config->create_empty ? "yes"
: "no");
+    fprintf(stderr, "Input pipe held open:        %12s\n", config->ignore_eof ? "yes"
: "no");
+    fprintf(stderr, "Input pipe:         %21s\n", config->inputpipe);
     fprintf(stderr, "Rotation file name: %21s\n", config->szLogRoot);
     fprintf(stderr, "Post-rotation prog: %21s\n", config->postrotate_prog);
@@ -529,9 +535,9 @@
     apr_pool_create(&status.pool, NULL);
     apr_getopt_init(&opt, status.pool, argc, argv);
-    while ((rv = apr_getopt(opt, "lL:p:ftvec", &c, &opt_arg)) == APR_SUCCESS) {
+    while ((rv = apr_getopt(opt, "lL:p:i:oftvec", &c, &opt_arg)) == APR_SUCCESS)
-    while ((rv = apr_getopt(opt, "lL:p:ftve", &c, &opt_arg)) == APR_SUCCESS) {
+    while ((rv = apr_getopt(opt, "lL:p:i:ftve", &c, &opt_arg)) == APR_SUCCESS) {
         switch (c) {
         case 'l':
@@ -543,6 +549,12 @@
         case 'p':
             config.postrotate_prog = opt_arg;
+        case 'i':
+            config.inputpipe = opt_arg;
+            break;
+        case 'o':
+            config.ignore_eof = 1;
+            break;
         case 'f':
             config.force_open = 1;
@@ -588,9 +600,25 @@
     config.use_strftime = (strchr(config.szLogRoot, '%') != NULL);
-    if (apr_file_open_stdin(&f_stdin, status.pool) != APR_SUCCESS) {
-        fprintf(stderr, "Unable to open stdin\n");
-        exit(1);
+    if (config.inputpipe) {
+        rv = apr_file_namedpipe_open(&f_stdin, config.inputpipe,
+                                     APR_FOPEN_READ |
+                                     (config.ignore_eof ? APR_FOPEN_WRITE : 0),
+                                     APR_FULL_NONBLOCK, status.pool);
+        if (rv != APR_SUCCESS) {
+            fprintf(stderr, "Unable to open pipe %s\n", config.inputpipe);
+            exit(1);
+        }
+    } else {
+        if (config.ignore_eof) {
+            usage(argv[0], "-o requires -i <inputpipe>");
+        }
+        if (apr_file_open_stdin(&f_stdin, status.pool) != APR_SUCCESS) {
+            fprintf(stderr, "Unable to open stdin\n");
+            exit(1);
+        }
     if (apr_file_open_stdout(&f_stdout, status.pool) != APR_SUCCESS) {
@@ -637,7 +665,11 @@
         if (pollret == APR_SUCCESS) {
             rv = apr_file_read(f_stdin, buf, &nRead);
             if (APR_STATUS_IS_EOF(rv)) {
-                break;
+                if (config.ignore_eof) {
+                    continue;
+                } else {
+                    break;
+                }
             else if (rv != APR_SUCCESS) {

GPG Key ID: 088335A9
GPG Key Fingerprint: 86CD 3297 7493 75BC F820  6715 F54F E648 0883 35A9

View raw message