httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mark <m...@node.to>
Subject Re: Sample code for IPC in modules
Date Wed, 05 May 2004 02:18:26 GMT
On May 4, 2004, at 8:49 PM, Sander Temme wrote:

> Hi all,
>
> Following some questions on the apache-modules list, I whipped up a 
> quick module for Apache 2.0 to hopefully demonstrate how it's done. 
> I'm including the module code below: please tell me whether I'm 
> smoking crack before I post this on apache-modules.
>
> Thanks for your thoughts,

A few notes, based on my experience using shm in 2.0 and observations 
of mod_auth_digest's and mod_ssl's implementation.

1)
No  attach logic is needed provided you keep the base address from 
apr_shm_create in a global var. It will get inherited when the children 
are forked. I separate shm handling out into a separate file that is 
linked in. It declares a static global base, and provides accessor 
functions to make that work cleanly.

Your attach logic should work, however it raises privilege issues 
because the children run as a different user (nobody or www, etc) the 
than the process running the create (root). I had problems when I was 
doing it that way, worked ok on freebsd and linux, but had strange 
behaviour on solaris. Attach is primarily designed to be used in cases 
where a totally separate process wants access to a piece of shared 
memory, not a forked one.

2)
Dettach is never needed. However, depending on desired results, it is 
usually desireable to perform a destroy when a HUP signal is sent, so 
that it gets created fresh by post_config

I've run into the strange errors under high load where newly forked 
children startup thinking they are attached to the inherited shm seg, 
but are in fact attached to some anonymous new segment. No error is 
produced, but obviously it's a catastrophic situation.

>
> Sander
>
> /*
> **  mod_example_ipc.c -- Apache sample example_ipc module
> **  [Autogenerated via ``apxs -n example_ipc -g'']
> **
> **  To play with this sample module first compile it into a
> **  DSO file and install it into Apache's modules directory
> **  by running:
> **
> **    $ apxs -c -i mod_example_ipc.c
> **
> **  Then activate it in Apache's httpd.conf file for instance
> **  for the URL /example_ipc in as follows:
> **
> **    #   httpd.conf
> **    LoadModule example_ipc_module modules/mod_example_ipc.so
> **    <Location /example_ipc>
> **    SetHandler example_ipc
> **    </Location>
> **
> **  Then after restarting Apache via
> **
> **    $ apachectl restart
> **
> **  The module allocates a counter in shared memory, which is 
> incremented
> **  by the request handler under a mutex. After installation, hit the 
> server
> **  with ab at various concurrency levels to see how mutex contention 
> affects
> **  server performance.
> */
>
> #include <sys/types.h>
> #include <unistd.h>
>
> #include "httpd.h"
> #include "http_config.h"
> #include "http_log.h"
> #include "http_protocol.h"
> #include "ap_config.h"
>
> #include "apr_strings.h"
>
> #define HTML_HEADER "<html>\n<head>\n<title>Mod_example_IPC Status

> Page " \
>                     "</title>\n</head>\n<body>\n<h1>Mod_example_IPC

> Status</h1>\n"
> #define HTML_FOOTER "</body>\n</html>\n"
>
> /* Number of microseconds to camp out on the mutex */
> #define CAMPOUT 10
> /* Maximum number of times we camp out before giving up */
> #define MAXCAMP 10
>
> apr_shm_t *exipc_shm;
> char *shmfilename;
> apr_global_mutex_t *exipc_mutex;
> char *mutexfilename;
>
> typedef struct exipc_data {
>     apr_uint64_t counter;
>     /* More fields if necessary */
> } exipc_data;
>
> /*
>  * This routine is called in the parent, so we'll set up the shared
>  * memory segment and mutex here.
>  */
>
> static int exipc_post_config(apr_pool_t *pconf, apr_pool_t *plog,
>                              apr_pool_t *ptemp, server_rec *s)
> {
>     void *data; /* These two help ensure that we only init once. */
>     const char *userdata_key = "example_ipc_init_module";
>     apr_status_t rs;
>     exipc_data *base;
>
>
>     /*
>      * The following checks if this routine has been called before.
>      * This is necessary because the parent process gets initialized
>      * a couple of times as the server starts up, and we don't want
>      * to create any more mutexes and shared memory segments than
>      * we're actually going to use.
>      */
>     apr_pool_userdata_get(&data, userdata_key, s->process->pool);
>     if (!data) {
>         apr_pool_userdata_set((const void *) 1, userdata_key,
>                               apr_pool_cleanup_null, s->process->pool);
>         return OK;
>     } /* Kilroy was here */
>
>     /* Create the shared memory segment */
>
>     /*
>      * Create a unique filename using our pid. This information is
>      * stashed in the global variable so the children inherit it.
>      * TODO get the location from the environment $TMPDIR or somesuch.
>      */
>     shmfilename = apr_psprintf(pconf, "/tmp/httpd_shm.%ld", (long 
> int)getpid());
>
>     /* Now create that segment */
>     rs = apr_shm_create(&exipc_shm, sizeof(exipc_data),
>                         (const char *) shmfilename, pconf);
>     if (rs != APR_SUCCESS) {
>         ap_log_error(APLOG_MARK, APLOG_ERR, rs, s,
>                      "Failed to create shared memory segment on file 
> %s",
>                      shmfilename);
>         return HTTP_INTERNAL_SERVER_ERROR;
>     }
>
>     /* Created it, now let's zero it out */
>     base = (exipc_data *)apr_shm_baseaddr_get(exipc_shm);
>     base->counter = 0;
>
>     /* Create global mutex */
>
>     /*
>      * Create another unique filename to lock upon. Note that
>      * depending on OS and locking mechanism of choice, the file
>      * may or may not be actually created.
>      */
>     mutexfilename = apr_psprintf(pconf, "/tmp/httpd_mutex.%ld",
>                                  (long int) getpid());
>
>     rs = apr_global_mutex_create(&exipc_mutex, (const char *) 
> mutexfilename,
>                                  APR_LOCK_DEFAULT, pconf);
>     if (rs != APR_SUCCESS) {
>         ap_log_error(APLOG_MARK, APLOG_ERR, rs, s,
>                      "Failed to create mutex on file %s",
>                      mutexfilename);
>         return HTTP_INTERNAL_SERVER_ERROR;
>     }
>
>     return OK;
> }
>
> /*
>  * This routine gets called when a child exits. It detaches from the
>  * shared memory segment. (is this necessary?)
>  */
>
> static apr_status_t exipc_child_exit(void *data)
> {
>     apr_status_t rs;
>
>     rs = apr_shm_detach(exipc_shm);
>     return rs;
> }
>
> /*
>  * This routine gets called when a child inits. We use it to attach
>  * to the shared memory segment, and reinitialize the mutex.
>  */
>
> static void exipc_child_init(apr_pool_t *p, server_rec *s)
> {
>     apr_status_t rs;
>
>     /*
>      * Attach to the shared memory segment. Note that we're
>      * reusing the global variable here: the data in that
>      * may have meaning only to the parent process that
>      * created the segment. We're identifying the segment
>      * through the filename (which we inherited from the parent.
>      */
>     rs = apr_shm_attach(&exipc_shm, (const char *) shmfilename, p);
>     if (rs != APR_SUCCESS) {
>          ap_log_error(APLOG_MARK, APLOG_CRIT, rs, s,
>                      "Failed to attach to shared memory segment on 
> file %s",
>                      shmfilename);
>          /* There's really nothing else we can do here, since
>           * This routine doesn't return a status. */
>          exit(1); /* Ugly, but what else? */
>      }
>
>      /*
>       * Re-open the mutex for the child. Note we're also reusing
>       * the mutex pointer global here.
>       */
>      rs = apr_global_mutex_child_init(&exipc_mutex,
>                                       (const char *) mutexfilename,
>                                       p);
>      if (rs != APR_SUCCESS) {
>          ap_log_error(APLOG_MARK, APLOG_CRIT, rs, s,
>                      "Failed to reopen mutex on file %s",
>                      shmfilename);
>          /* There's really nothing else we can do here, since
>           * This routine doesn't return a status. */
>          exit(1); /* Ugly, but what else? */
>      }
>
>     /* Install the exit routine as cleanup for the pool. There
>      * is no hook for this.
>      */
>     apr_pool_cleanup_register(p, s, exipc_child_exit, 
> exipc_child_exit);
> }
>
> /* The sample content handler */
> static int exipc_handler(request_rec *r)
> {
>     int gotlock = 0;
>     int camped;
>     apr_time_t startcamp;
>     apr_int64_t timecamped;
>     apr_status_t rs;
>     exipc_data *base;
>
>     if (strcmp(r->handler, "example_ipc")) {
>         return DECLINED;
>     }
>
>     /*
>      * The main function of the handler, aside from sending the
>      * status page to the client, is to increment the counter in
>      * the shared memory segment. This action needs to be mutexed
>      * out using the global mutex.
>      */
>
>     /* First, acquire the lock */
>     for (camped = 0, timecamped = 0; camped < MAXCAMP; camped++) {
>         rs = apr_global_mutex_trylock(exipc_mutex);
>         if (APR_STATUS_IS_EBUSY(rs)) {
>             apr_sleep(CAMPOUT);
>         } else if (APR_STATUS_IS_SUCCESS(rs)) {
>             gotlock = 1;
>             break; /* Get out of the loop */
>         } else if (APR_STATUS_IS_ENOTIMPL(rs)) {
>             /* If it's not implemented, just hang in the mutex. */
>             startcamp = apr_time_now();
>             rs = apr_global_mutex_lock(exipc_mutex);
>             timecamped = (apr_int64_t) (apr_time_now() - startcamp);
>             if (APR_STATUS_IS_SUCCESS(rs)) {
>                 gotlock = 1;
>                 break;
>             } else {
>                 /* Some error, log and bail */
>                 ap_log_error(APLOG_MARK, APLOG_ERR, rs, r->server,
>                              "Child %ld failed to acquire lock",
>                              (long int)getpid());
>                 return HTTP_INTERNAL_SERVER_ERROR;
>             }
>         } else {
>             /* Some other error, log and bail */
>             ap_log_error(APLOG_MARK, APLOG_ERR, rs, r->server,
>                          "Child %ld failed to try and acquire lock",
>                          (long int)getpid());
>             return HTTP_INTERNAL_SERVER_ERROR;
>
>         }
>         timecamped += CAMPOUT;
>         ap_log_error(APLOG_MARK, APLOG_NOERRNO | APLOG_NOTICE,
>                      0, r->server, "Child %ld camping out on mutex for 
> %d microseconds",
>                      (long int) getpid(), camped * CAMPOUT);
>     }
>
>     /* Sleep for a millisecond to make it a little harder for
>      * httpd children to acquire the lock.
>      */
>     apr_sleep(1000);
>
>     r->content_type = "text/html";
>
>     if (!r->header_only) {
>         ap_rputs(HTML_HEADER, r);
>         if (gotlock) {
>             /* Increment the counter */
>             base = (exipc_data *)apr_shm_baseaddr_get(exipc_shm);
>             base->counter++;
>             /* Send a page with our pid and the new value of the 
> counter. */
>             ap_rprintf(r, "<p>Lock acquired after camping out for %ld 
> microseoncds.</p>\n",
>                        (long int) timecamped);
>             ap_rputs("<table border=\"1\">\n", r);
>             ap_rprintf(r, "<tr><td>Child pid:</td><td>%d</td></tr>\n",
>                        (int) getpid());
>             ap_rprintf(r, "<tr><td>Counter:</td><td>%u</td></tr>\n",
>                        (unsigned int)base->counter);
>             ap_rputs("</table>\n", r);
>         } else {
>             /*
>              * Send a page saying that we couldn't get the lock. Don't 
> say
>              * what the counter is, because without the lock the value 
> could
>              * race.
>              */
>              ap_rprintf(r, "<p>Child %d failed to acquire lock "
>                         "after camping out for %d microseconds.</p>\n",
>                         (int) getpid(), (int) timecamped);
>         }
>         ap_rputs(HTML_FOOTER, r);
>     } /* r->header_only */
>
>     /* Release the lock */
>     if (gotlock)
>         rs = apr_global_mutex_unlock(exipc_mutex);
>     /* Swallowing the result because what are we going to do with it at
>      * this stage?
>      */
>
>     return OK;
> }
>
> static void exipc_register_hooks(apr_pool_t *p)
> {
>     ap_hook_post_config(exipc_post_config, NULL, NULL, 
> APR_HOOK_MIDDLE);
>     ap_hook_child_init(exipc_child_init, NULL, NULL, APR_HOOK_MIDDLE);
>     ap_hook_handler(exipc_handler, NULL, NULL, APR_HOOK_MIDDLE);
> }
>
> /* Dispatch list for API hooks */
> module AP_MODULE_DECLARE_DATA example_ipc_module = {
>     STANDARD20_MODULE_STUFF,
>     NULL,                  /* create per-dir    config structures */
>     NULL,                  /* merge  per-dir    config structures */
>     NULL,                  /* create per-server config structures */
>     NULL,                  /* merge  per-server config structures */
>     NULL,                  /* table of config file commands       */
>     exipc_register_hooks   /* register hooks                      */
> };
>
>
> -- 
> sander@temme.net              http://www.temme.net/sander/
> PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF


Mime
View raw message