httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Galić <i.ga...@brainsware.org>
Subject Re: rotatelogs input from named pipe
Date Mon, 24 Dec 2012 12:48:02 GMT

Hey folks,

has anyone followed up on Chris' patch?

----- Original Message -----
> Hi --
> 
>    I recently needed to wire up httpd's rotatelogs to a named pipe (a
>    FIFO)
> instead of to stdin.  For context, we have a non-httpd server process
> configured to write to the pipe instead of a log file; further, this
> server likes to intermittently open, write to, and close its various
> log files.
> 
>    I wanted/needed to open the pipe in non-blocking mode, so
>    rotatelogs
> would start right away (we're using rotatelogs -cf) instead of
> blocking
> until a writer process happened to open the pipe to write some log
> data.
> 
>    Also, because the server process repeatedly opens and closes its
> log files/pipes, a normal reader will exit on EOF after the first
> close,
> and therefore miss all subsequent writes.
> 
>    I played around for some time with variants on "tail -f pipe |
>    rotatelogs"
> and "cat pipe | rotatelogs" but both have problems.  The tail -f
> seems
> to miss data, especially at first; I can't explain that but it's what
> I observed.  The cat command works to capture all data while running,
> but it blocks at startup and (worse for us) exits on the first EOF.
> 
>    What I really wanted was a simpler command line, passing the pipe
>    to
> rotatelogs directly, and having it (a) open in non-blocking mode so
> the -cf options are effective right away, and (b) ignore EOFs until I
> kill
> it explicitly.  The latter is ungainly but that's how the server also
> has to be stopped, via kill -TERM, so it's acceptable in this case
> for us.
> 
>    First I puzzled over how to open a pipe in non-blocking mode from
>    the
> start using APR, and ended up with a set of patches for that; see
> http://marc.info/?l=apr-dev&m=135586687408393&w=2 for the details,
> but the gist is an apr_file_namedpipe_open() function to do this.
> Probably not sufficiently full of trademark portable goodness, but
> effective for the moment.
> 
>    Next the following patch adds a "-i inputpipe [-o]" set of
>    options,
> when possible, so that rotatelogs will listen to the pipe instead of
> stdin and will ignore EOFs.
> 
>    I'm posting both the APR and httpd patches because I thought it
>    was
> an interesting issue, and I suspect there are alternate and perhaps
> better approaches; certainly the APR patch isn't ideal.  But after
> a long time spent digging into the source code for tail and cat and
> trying many variants, this now seems to work smoothly for us, at
> least.
> 
> Chris.
> 
> ===================================================================
> --- support/rotatelogs.c.orig	2012-12-18 13:44:02.000000000 -0800
> +++ support/rotatelogs.c	2012-12-18 14:11:56.000000000 -0800
> @@ -67,6 +67,8 @@
>      int truncate;
>      const char *linkfile;
>      const char *postrotate_prog;
> +    const char *inputpipe;
> +    int ignore_eof;
>  #if APR_FILES_AS_SOCKETS
>      int create_empty;
>  #endif
> @@ -101,9 +103,9 @@
>      }
>      fprintf(stderr,
>  #if APR_FILES_AS_SOCKETS
> -            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t]
> [-e] [-c] <logfile> "
> +            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i
> inputpipe [-o]] [-f] [-t] [-e] [-c] <logfile> "
>  #else
> -            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t]
> [-e] <logfile> "
> +            "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i
> inputpipe] [-f] [-t] [-e] <logfile> "
>  #endif
>              "{<rotation time in seconds>|<rotation size>(B|K|M|G)} "
>              "[offset minutes from UTC]\n\n",
> @@ -137,7 +139,9 @@
>              "  -f       Force opening of log on program start.\n"
>              "  -t       Truncate logfile instead of rotating, tail
>              friendly.\n"
>              "  -e       Echo log to stdout for further
>              processing.\n"
> +            "  -i path  Use specified named pipe as input.\n"
>  #if APR_FILES_AS_SOCKETS
> +            "  -o       Hold input pipe open when writers exit (with
> -i only).\n"
>              "  -c       Create log even if it is empty.\n"
>  #endif
>              "\n"
> @@ -194,7 +198,9 @@
>      fprintf(stderr, "Rotation verbose:            %12s\n",
>      config->verbose ? "yes" : "no");
>  #if APR_FILES_AS_SOCKETS
>      fprintf(stderr, "Rotation create empty logs:  %12s\n",
>      config->create_empty ? "yes" : "no");
> +    fprintf(stderr, "Input pipe held open:        %12s\n",
> config->ignore_eof ? "yes" : "no");
>  #endif
> +    fprintf(stderr, "Input pipe:         %21s\n",
> config->inputpipe);
>      fprintf(stderr, "Rotation file name: %21s\n",
>      config->szLogRoot);
>      fprintf(stderr, "Post-rotation prog: %21s\n",
>      config->postrotate_prog);
>  }
> @@ -529,9 +535,9 @@
>      apr_pool_create(&status.pool, NULL);
>      apr_getopt_init(&opt, status.pool, argc, argv);
>  #if APR_FILES_AS_SOCKETS
> -    while ((rv = apr_getopt(opt, "lL:p:ftvec", &c, &opt_arg)) ==
> APR_SUCCESS) {
> +    while ((rv = apr_getopt(opt, "lL:p:i:oftvec", &c, &opt_arg)) ==
> APR_SUCCESS) {
>  #else
> -    while ((rv = apr_getopt(opt, "lL:p:ftve", &c, &opt_arg)) ==
> APR_SUCCESS) {
> +    while ((rv = apr_getopt(opt, "lL:p:i:ftve", &c, &opt_arg)) ==
> APR_SUCCESS) {
>  #endif
>          switch (c) {
>          case 'l':
> @@ -543,6 +549,12 @@
>          case 'p':
>              config.postrotate_prog = opt_arg;
>              break;
> +        case 'i':
> +            config.inputpipe = opt_arg;
> +            break;
> +        case 'o':
> +            config.ignore_eof = 1;
> +            break;
>          case 'f':
>              config.force_open = 1;
>              break;
> @@ -588,9 +600,25 @@
>  
>      config.use_strftime = (strchr(config.szLogRoot, '%') != NULL);
>  
> -    if (apr_file_open_stdin(&f_stdin, status.pool) != APR_SUCCESS) {
> -        fprintf(stderr, "Unable to open stdin\n");
> -        exit(1);
> +    if (config.inputpipe) {
> +        rv = apr_file_namedpipe_open(&f_stdin, config.inputpipe,
> +                                     APR_FOPEN_READ |
> +                                     (config.ignore_eof ?
> APR_FOPEN_WRITE : 0),
> +                                     APR_FULL_NONBLOCK,
> status.pool);
> +
> +        if (rv != APR_SUCCESS) {
> +            fprintf(stderr, "Unable to open pipe %s\n",
> config.inputpipe);
> +            exit(1);
> +        }
> +    } else {
> +        if (config.ignore_eof) {
> +            usage(argv[0], "-o requires -i <inputpipe>");
> +        }
> +
> +        if (apr_file_open_stdin(&f_stdin, status.pool) !=
> APR_SUCCESS) {
> +            fprintf(stderr, "Unable to open stdin\n");
> +            exit(1);
> +        }
>      }
>  
>      if (apr_file_open_stdout(&f_stdout, status.pool) != APR_SUCCESS)
>      {
> @@ -637,7 +665,11 @@
>          if (pollret == APR_SUCCESS) {
>              rv = apr_file_read(f_stdin, buf, &nRead);
>              if (APR_STATUS_IS_EOF(rv)) {
> -                break;
> +                if (config.ignore_eof) {
> +                    continue;
> +                } else {
> +                    break;
> +                }
>              }
>              else if (rv != APR_SUCCESS) {
>                  exit(3);
> ===================================================================
> 
> --
> GPG Key ID: 088335A9
> GPG Key Fingerprint: 86CD 3297 7493 75BC F820  6715 F54F E648 0883
> 35A9
> 
> 

-- 
Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 6880 4155 74BD FD7C B515  2EA5 4B1D 9E08 A097 C9AE


Mime
View raw message