httpd-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From minf...@apache.org
Subject svn commit: r917013 - in /httpd/httpd/branches/2.2.x: CHANGES STATUS docs/manual/mod/mod_cache.xml modules/cache/cache_storage.c modules/cache/cache_util.c modules/cache/mod_cache.c modules/cache/mod_cache.h
Date Sat, 27 Feb 2010 18:54:41 GMT
Author: minfrin
Date: Sat Feb 27 18:54:40 2010
New Revision: 917013

URL: http://svn.apache.org/viewvc?rev=917013&view=rev
Log:
mod_cache: Introduce the thundering herd lock, a mechanism to keep
the flood of requests at bay that strike a backend webserver as
a cached entity goes stale.
+1: minfrin, jim, pgollucci

Modified:
    httpd/httpd/branches/2.2.x/CHANGES
    httpd/httpd/branches/2.2.x/STATUS
    httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml
    httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c
    httpd/httpd/branches/2.2.x/modules/cache/cache_util.c
    httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c
    httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h

Modified: httpd/httpd/branches/2.2.x/CHANGES
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/CHANGES?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/CHANGES [utf-8] (original)
+++ httpd/httpd/branches/2.2.x/CHANGES [utf-8] Sat Feb 27 18:54:40 2010
@@ -9,6 +9,10 @@
      access control is still vulnerable, unless using OpenSSL >= 0.9.8l.
      [Joe Orton, Ruediger Pluem, Hartmut Keil <Hartmut.Keil adnovum.ch>]
 
+  *) mod_cache: Introduce the thundering herd lock, a mechanism to keep
+     the flood of requests at bay that strike a backend webserver as
+     a cached entity goes stale. [Graham Leggett]
+
   *) mod_proxy_http: Make sure that when an ErrorDocument is served
      from a reverse proxied URL, that the subrequest respects the status
      of the original request. This brings the behaviour of proxy_handler

Modified: httpd/httpd/branches/2.2.x/STATUS
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/STATUS (original)
+++ httpd/httpd/branches/2.2.x/STATUS Sat Feb 27 18:54:40 2010
@@ -87,19 +87,6 @@
 PATCHES ACCEPTED TO BACKPORT FROM TRUNK:
   [ start all new proposals below, under PATCHES PROPOSED. ]
 
- * mod_cache: Introduce the thundering herd lock, a mechanism to keep
-              the flood of requests at bay that strike a backend webserver as
-              a cached entity goes stale.
-   Trunk Patch: http://svn.apache.org/viewvc?rev=808212&view=rev
-                http://svn.apache.org/viewvc?rev=808220&view=rev
-                http://svn.apache.org/viewvc?rev=808649&view=rev
-                http://svn.apache.org/viewvc?rev=808652&view=rev
-                http://svn.apache.org/viewvc?rev=808656&view=rev
-                http://svn.apache.org/viewvc?rev=809440&view=rev
-                http://svn.apache.org/viewvc?rev=809665&view=rev
-   2.2.x Patch: http://people.apache.org/~minfrin/httpd-cache-thundering.patch
-   +1: minfrin, jim, pgollucci
-
   * rewrite_log: remove locking
     trunk patch: http://svn.apache.org/viewvc?view=revision&revision=783734
 

Modified: httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml (original)
+++ httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml Sat Feb 27 18:54:40 2010
@@ -125,6 +125,70 @@
     </example>
 </section>
 
+<section id="thunderingherd"><title>Avoiding the Thundering Herd</title>
+  <p>When a cached entry becomes stale, <module>mod_cache</module> will
submit
+  a conditional request to the backend, which is expected to confirm whether the
+  cached entry is still fresh, and send an updated entity if not.</p>
+  <p>A small but finite amount of time exists between the time the cached entity
+  becomes stale, and the time the stale entity is fully refreshed. On a busy
+  server, a significant number of requests might arrive during this time, and
+  cause a <strong>thundering herd</strong> of requests to strike the backend
+  suddenly and unpredicably.</p>
+  <p>To keep the thundering herd at bay, the <directive>CacheLock</directive>
+  directive can be used to define a directory in which locks are created for
+  URLs <strong>in flight</strong>. The lock is used as a <strong>hint</strong>
+  by other requests to either suppress an attempt to cache (someone else has
+  gone to fetch the entity), or to indicate that a stale entry is being refreshed
+  (stale content will be returned in the mean time).
+  </p>
+  <section>
+    <title>Initial caching of an entry</title>
+    <p>When an entity is cached for the first time, a lock will be created for the
+    entity until the response has been fully cached. During the lifetime of the
+    lock, the cache will suppress the second and subsequent attempt to cache the
+    same entity. While this doesn't hold back the thundering herd, it does stop
+    the cache attempting to cache the same entity multiple times simultaneously.
+    </p>
+  </section>
+  <section>
+    <title>Refreshment of a stale entry</title>
+    <p>When an entity reaches its freshness lifetime and becomes stale, a lock
+    will be created for the entity until the response has either been confirmed as
+    still fresh, or replaced by the backend. During the lifetime of the lock, the
+    second and subsequent incoming request will cause stale data to be returned,
+    and the thundering herd is kept at bay.</p>
+  </section>
+  <section>
+    <title>Locks and Cache-Control: no-cache</title>
+    <p>Locks are used as a <strong>hint only</strong> to enable the cache
to be
+    more gentle on backend servers, however the lock can be overridden if necessary.
+    If the client sends a request with a Cache-Control header forcing a reload, any
+    lock that may be present will be ignored, and the client's request will be
+    honoured immediately and the cached entry refreshed.</p>
+    <p>As a further safety mechanism, locks have a configurable maximum age.
+    Once this age has been reached, the lock is removed, and a new request is
+    given the opportunity to create a new lock. This maximum age can be set using
+    the <directive>CacheLockMaxAge</directive> directive, and defaults to 5
+    seconds.
+    </p>
+  </section>
+  <section>
+    <title>Example configuration</title>
+    <example><title>Enabling the cache lock</title>
+      #<br />
+      # Enable the cache lock<br />
+      #<br />
+      &lt;IfModule mod_cache.c&gt;<br />
+      <indent>
+        CacheLock on<br />
+        CacheLockPath /tmp/mod_cache-lock<br />
+        CacheLockMaxAge 5<br />
+      </indent>
+      &lt;/IfModule&gt;
+    </example>
+  </section>
+</section>
+
 <directivesynopsis>
 <name>CacheEnable</name>
 <description>Enable caching of specified URLs using a specified storage
@@ -513,4 +577,66 @@
 <seealso><directive module="mod_cache">CacheIgnoreCacheControl</directive></seealso>
 <seealso><directive module="mod_cache">CacheStorePrivate</directive></seealso>
 </directivesynopsis>
+
+<directivesynopsis>
+<name>CacheLock</name>
+<description>Enable the thundering herd lock.</description>
+<syntax>CacheLock <var>on|off</var></syntax>
+<default>CacheLock off</default>
+<contextlist><context>server config</context><context>virtual host</context>
+</contextlist>
+
+<usage>
+  <p>The <directive>CacheLock</directive> directive enables the thundering
herd lock
+  for the given URL space.</p>
+  
+  <p>In a minimal configuration the following directive is all that is needed to
+  enable the thundering herd lock in the default system temp directory.</p>
+
+  <example>
+    # Enable chache lock<br />
+    CacheLock on<br /><br />
+  </example>
+
+</usage>
+</directivesynopsis>
+
+<directivesynopsis>
+<name>CacheLockPath</name>
+<description>Set the lock path directory.</description>
+<syntax>CacheLockPath <var>directory</var></syntax>
+<default>CacheLockPath /tmp/mod_cache-lock</default>
+<contextlist><context>server config</context><context>virtual host</context>
+</contextlist>
+    
+<usage>
+  <p>The <directive>CacheLockPath</directive> directive allows you to specify
the
+  directory in which the locks are created. By default, the system's temporary
+  folder is used. Locks consist of empty files that only exist for stale URLs
+  in flight, so is significantly less resource intensive than the traditional
+  disk cache.</p>
+
+</usage>
+</directivesynopsis>
+  
+<directivesynopsis>
+<name>CacheLockMaxAge</name>
+<description>Set the maximum possible age of a cache lock.</description>
+<syntax>CacheLockMaxAge <var>integer</var></syntax>
+<default>CacheLockMaxAge 5</default>
+<contextlist><context>server config</context><context>virtual host</context>
+</contextlist>
+    
+<usage>
+  <p>The <directive>CacheLockMaxAge</directive> directive specifies the
maximum
+  age of any cache lock.</p>
+  
+  <p>A lock older than this value in seconds will be ignored, and the next
+  incoming request will be given the opportunity to re-establish the lock.
+  This mechanism prevents a slow client taking an excessively long time to refresh
+  an entity.</p>
+  
+</usage>
+</directivesynopsis>
+  
 </modulesynopsis>

Modified: httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c Sat Feb 27 18:54:40 2010
@@ -37,7 +37,7 @@
 
     /* Remove the stale cache entry if present. If not, we're
      * being called from outside of a request; remove the
-     * non-stalle handle.
+     * non-stale handle.
      */
     h = cache->stale_handle ? cache->stale_handle : cache->handle;
     if (!h) {

Modified: httpd/httpd/branches/2.2.x/modules/cache/cache_util.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/cache_util.c?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/cache_util.c (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/cache_util.c Sat Feb 27 18:54:40 2010
@@ -22,6 +22,8 @@
 
 /* -------------------------------------------------------------- */
 
+extern APR_OPTIONAL_FN_TYPE(ap_cache_generate_key) *cache_generate_key;
+
 extern module AP_MODULE_DECLARE_DATA cache_module;
 
 /* Determine if "url" matches the hostname, scheme and port and path
@@ -164,9 +166,182 @@
     return apr_time_sec(current_age);
 }
 
+/**
+ * Try obtain a cache wide lock on the given cache key.
+ *
+ * If we return APR_SUCCESS, we obtained the lock, and we are clear to
+ * proceed to the backend. If we return APR_EEXISTS, then the lock is
+ * already locked, someone else has gone to refresh the backend data
+ * already, so we must return stale data with a warning in the mean
+ * time. If we return anything else, then something has gone pear
+ * shaped, and we allow the request through to the backend regardless.
+ *
+ * This lock is created from the request pool, meaning that should
+ * something go wrong and the lock isn't deleted on return of the
+ * request headers from the backend for whatever reason, at worst the
+ * lock will be cleaned up when the request dies or finishes.
+ *
+ * If something goes truly bananas and the lock isn't deleted when the
+ * request dies, the lock will be trashed when its max-age is reached,
+ * or when a request arrives containing a Cache-Control: no-cache. At
+ * no point is it possible for this lock to permanently deny access to
+ * the backend.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_try_lock(cache_server_conf *conf,
+        request_rec *r, char *key) {
+    apr_status_t status;
+    const char *lockname;
+    const char *path;
+    char dir[5];
+    apr_time_t now = apr_time_now();
+    apr_finfo_t finfo;
+    apr_file_t *lockfile;
+    void *dummy;
+
+    finfo.mtime = 0;
+
+    if (!conf || !conf->lock || !conf->lockpath) {
+        /* no locks configured, leave */
+        return APR_SUCCESS;
+    }
+
+    /* lock already obtained earlier? if so, success */
+    apr_pool_userdata_get(&dummy, CACHE_LOCKFILE_KEY, r->pool);
+    if (dummy) {
+        return APR_SUCCESS;
+    }
+
+    /* create the key if it doesn't exist */
+    if (!key) {
+        cache_generate_key(r, r->pool, &key);
+    }
+
+    /* create a hashed filename from the key, and save it for later */
+    lockname = ap_cache_generate_name(r->pool, 0, 0, key);
+
+    /* lock files represent discrete just-went-stale URLs "in flight", so
+     * we support a simple two level directory structure, more is overkill.
+     */
+    dir[0] = '/';
+    dir[1] = lockname[0];
+    dir[2] = '/';
+    dir[3] = lockname[1];
+    dir[4] = 0;
+
+    /* make the directories */
+    path = apr_pstrcat(r->pool, conf->lockpath, dir, NULL);
+    if (APR_SUCCESS != (status = apr_dir_make_recursive(path,
+            APR_UREAD|APR_UWRITE|APR_UEXECUTE, r->pool))) {
+        ap_log_error(APLOG_MARK, APLOG_ERR, status, r->server,
+                     "Could not create a cache lock directory: %s",
+                     path);
+        return status;
+    }
+    lockname = apr_pstrcat(r->pool, path, "/", lockname, NULL);
+    apr_pool_userdata_set(lockname, CACHE_LOCKNAME_KEY, NULL, r->pool);
+
+    /* is an existing lock file too old? */
+    status = apr_stat(&finfo, lockname,
+                APR_FINFO_MTIME | APR_FINFO_NLINK, r->pool);
+    if (!(APR_STATUS_IS_ENOENT(status)) && APR_SUCCESS != status) {
+        ap_log_error(APLOG_MARK, APLOG_ERR, APR_EEXIST, r->server,
+                     "Could not stat a cache lock file: %s",
+                     lockname);
+        return status;
+    }
+    if (APR_SUCCESS == status && ((now - finfo.mtime) > conf->lockmaxage)
+            || (now < finfo.mtime)) {
+        ap_log_error(APLOG_MARK, APLOG_INFO, status, r->server,
+                     "Cache lock file for '%s' too old, removing: %s",
+                     r->uri, lockname);
+        apr_file_remove(lockname, r->pool);
+    }
+
+    /* try obtain a lock on the file */
+    if (APR_SUCCESS == (status = apr_file_open(&lockfile, lockname,
+            APR_WRITE | APR_CREATE | APR_EXCL | APR_DELONCLOSE,
+            APR_UREAD | APR_UWRITE, r->pool))) {
+        apr_pool_userdata_set(lockfile, CACHE_LOCKFILE_KEY, NULL, r->pool);
+    }
+    return status;
+
+}
+
+/**
+ * Remove the cache lock, if present.
+ *
+ * First, try to close the file handle, whose delete-on-close should
+ * kill the file. Otherwise, just delete the file by name.
+ *
+ * If no lock name has yet been calculated, do the calculation of the
+ * lock name first before trying to delete the file.
+ *
+ * If an optional bucket brigade is passed, the lock will only be
+ * removed if the bucket brigade contains an EOS bucket.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_remove_lock(cache_server_conf *conf,
+        request_rec *r, char *key, apr_bucket_brigade *bb) {
+    void *dummy;
+    const char *lockname;
+
+    if (!conf || !conf->lock || !conf->lockpath) {
+        /* no locks configured, leave */
+        return APR_SUCCESS;
+    }
+    if (bb) {
+        apr_bucket *e;
+        int eos_found = 0;
+        for (e = APR_BRIGADE_FIRST(bb);
+             e != APR_BRIGADE_SENTINEL(bb);
+             e = APR_BUCKET_NEXT(e))
+        {
+            if (APR_BUCKET_IS_EOS(e)) {
+                eos_found = 1;
+                break;
+            }
+        }
+        if (!eos_found) {
+            /* no eos found in brigade, don't delete anything just yet,
+             * we are not done.
+             */
+            return APR_SUCCESS;
+        }
+    }
+    apr_pool_userdata_get(&dummy, CACHE_LOCKFILE_KEY, r->pool);
+    if (dummy) {
+        return apr_file_close((apr_file_t *)dummy);
+    }
+    apr_pool_userdata_get(&dummy, CACHE_LOCKNAME_KEY, r->pool);
+    lockname = (const char *)dummy;
+    if (!lockname) {
+        char dir[5];
+
+        /* create the key if it doesn't exist */
+        if (!key) {
+            cache_generate_key(r, r->pool, &key);
+        }
+
+        /* create a hashed filename from the key, and save it for later */
+        lockname = ap_cache_generate_name(r->pool, 0, 0, key);
+
+        /* lock files represent discrete just-went-stale URLs "in flight", so
+         * we support a simple two level directory structure, more is overkill.
+         */
+        dir[0] = '/';
+        dir[1] = lockname[0];
+        dir[2] = '/';
+        dir[3] = lockname[1];
+        dir[4] = 0;
+
+        lockname = apr_pstrcat(r->pool, conf->lockpath, dir, "/", lockname, NULL);
+    }
+    return apr_file_remove(lockname, r->pool);
+}
+
 CACHE_DECLARE(int) ap_cache_check_freshness(cache_handle_t *h,
                                             request_rec *r)
 {
+    apr_status_t status;
     apr_int64_t age, maxage_req, maxage_cresp, maxage, smaxage, maxstale;
     apr_int64_t minfresh;
     const char *cc_cresp, *cc_req;
@@ -176,6 +351,7 @@
     char *val;
     apr_time_t age_c = 0;
     cache_info *info = &(h->cache_obj->info);
+    const char *warn_head;
     cache_server_conf *conf =
       (cache_server_conf *)ap_get_module_config(r->server->module_config,
                                                 &cache_module);
@@ -338,7 +514,6 @@
         ((smaxage == -1) && (maxage == -1) &&
          (info->expire != APR_DATE_BAD) &&
          (age < (apr_time_sec(info->expire - info->date) + maxstale - minfresh))))
{
-        const char *warn_head;
 
         warn_head = apr_table_get(h->resp_hdrs, "Warning");
 
@@ -361,7 +536,7 @@
         }
         /*
          * If none of Expires, Cache-Control: max-age, or Cache-Control:
-         * s-maxage appears in the response, and the respose header age
+         * s-maxage appears in the response, and the response header age
          * calculated is more than 24 hours add the warning 113
          */
         if ((maxage_cresp == -1) && (smaxage == -1) &&
@@ -380,7 +555,71 @@
         return 1;    /* Cache object is fresh (enough) */
     }
 
-    return 0;        /* Cache object is stale */
+    /*
+     * At this point we are stale, but: if we are under load, we may let
+     * a significant number of stale requests through before the first
+     * stale request successfully revalidates itself, causing a sudden
+     * unexpected thundering herd which in turn brings angst and drama.
+     *
+     * So.
+     *
+     * We want the first stale request to go through as normal. But the
+     * second and subsequent request, we must pretend to be fresh until
+     * the first request comes back with either new content or confirmation
+     * that the stale content is still fresh.
+     *
+     * To achieve this, we create a very simple file based lock based on
+     * the key of the cached object. We attempt to open the lock file with
+     * exclusive write access. If we succeed, woohoo! we're first, and we
+     * follow the stale path to the backend server. If we fail, oh well,
+     * we follow the fresh path, and avoid being a thundering herd.
+     *
+     * The lock lives only as long as the stale request that went on ahead.
+     * If the request succeeds, the lock is deleted. If the request fails,
+     * the lock is deleted, and another request gets to make a new lock
+     * and try again.
+     *
+     * At any time, a request marked "no-cache" will force a refresh,
+     * ignoring the lock, ensuring an extended lockout is impossible.
+     *
+     * A lock that exceeds a maximum age will be deleted, and another
+     * request gets to make a new lock and try again.
+     */
+    status = ap_cache_try_lock(conf, r, (char *)h->cache_obj->key);
+    if (APR_SUCCESS == status) {
+        /* we obtained a lock, follow the stale path */
+        ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
+                     "Cache lock obtained for stale cached URL, "
+                     "revalidating entry: %s",
+                     r->unparsed_uri);
+        return 0;
+    }
+    else if (APR_EEXIST == status) {
+        /* lock already exists, return stale data anyway, with a warning */
+        ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
+                     "Cache already locked for stale cached URL, "
+                     "pretend it is fresh: %s",
+                     r->unparsed_uri);
+
+        /* make sure we don't stomp on a previous warning */
+        warn_head = apr_table_get(h->resp_hdrs, "Warning");
+        if ((warn_head == NULL) ||
+            ((warn_head != NULL) && (ap_strstr_c(warn_head, "110") == NULL))) {
+            apr_table_merge(h->resp_hdrs, "Warning",
+                        "110 Response is stale");
+        }
+
+        return 1;
+    }
+    else {
+        /* some other error occurred, just treat the object as stale */
+        ap_log_error(APLOG_MARK, APLOG_DEBUG, status, r->server,
+                     "Attempt to obtain a cache lock for stale "
+                     "cached URL failed, revalidating entry anyway: %s",
+                     r->unparsed_uri);
+        return 0;
+    }
+
 }
 
 /*

Modified: httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c Sat Feb 27 18:54:40 2010
@@ -114,40 +114,56 @@
         if (rv == DECLINED) {
             if (!lookup) {
 
-                /*
-                 * Add cache_save filter to cache this request. Choose
-                 * the correct filter by checking if we are a subrequest
-                 * or not.
+                /* try to obtain a cache lock at this point. if we succeed,
+                 * we are the first to try and cache this url. if we fail,
+                 * it means someone else is already trying to cache this
+                 * url, and we should just let the request through to the
+                 * backend without any attempt to cache. this stops
+                 * duplicated simultaneous attempts to cache an entity.
                  */
-                if (r->main) {
-                    ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
-                                 r->server,
-                                 "Adding CACHE_SAVE_SUBREQ filter for %s",
-                                 r->uri);
-                    ap_add_output_filter_handle(cache_save_subreq_filter_handle,
-                                                NULL, r, r->connection);
+                rv = ap_cache_try_lock(conf, r, NULL);
+                if (APR_SUCCESS == rv) {
+
+                    /*
+                     * Add cache_save filter to cache this request. Choose
+                     * the correct filter by checking if we are a subrequest
+                     * or not.
+                     */
+                    if (r->main) {
+                        ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
+                                r->server,
+                                "Adding CACHE_SAVE_SUBREQ filter for %s",
+                                r->uri);
+                        ap_add_output_filter_handle(cache_save_subreq_filter_handle,
+                                NULL, r, r->connection);
+                    }
+                    else {
+                        ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
+                                r->server, "Adding CACHE_SAVE filter for %s",
+                                r->uri);
+                        ap_add_output_filter_handle(cache_save_filter_handle,
+                                NULL, r, r->connection);
+                    }
+
+                    ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, r->server,
+                            "Adding CACHE_REMOVE_URL filter for %s",
+                            r->uri);
+
+                    /* Add cache_remove_url filter to this request to remove a
+                     * stale cache entry if needed. Also put the current cache
+                     * request rec in the filter context, as the request that
+                     * is available later during running the filter maybe
+                     * different due to an internal redirect.
+                     */
+                    cache->remove_url_filter =
+                        ap_add_output_filter_handle(cache_remove_url_filter_handle,
+                                cache, r, r->connection);
                 }
                 else {
-                    ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
-                                 r->server, "Adding CACHE_SAVE filter for %s",
-                                 r->uri);
-                    ap_add_output_filter_handle(cache_save_filter_handle,
-                                                NULL, r, r->connection);
+                    ap_log_error(APLOG_MARK, APLOG_DEBUG, rv,
+                                 r->server, "Cache locked for url, not caching "
+                                 "response: %s", r->uri);
                 }
-
-                ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, r->server,
-                             "Adding CACHE_REMOVE_URL filter for %s",
-                             r->uri);
-
-                /* Add cache_remove_url filter to this request to remove a
-                 * stale cache entry if needed. Also put the current cache
-                 * request rec in the filter context, as the request that
-                 * is available later during running the filter maybe
-                 * different due to an internal redirect.
-                 */
-                cache->remove_url_filter =
-                    ap_add_output_filter_handle(cache_remove_url_filter_handle,
-                                                cache, r, r->connection);
             }
             else {
                 if (cache->stale_headers) {
@@ -166,7 +182,7 @@
             /* error */
             ap_log_error(APLOG_MARK, APLOG_ERR, rv, r->server,
                          "cache: error returned while checking for cached "
-                         "file by %s cache", cache->provider_name);
+                         "file by '%s' cache", cache->provider_name);
         }
         return DECLINED;
     }
@@ -313,6 +329,10 @@
  *        Check to see if we *can* save this particular response.
  *        If we can, call cache_create_entity() and save the headers and body
  *   Finally, pass the data to the next filter (the network or whatever)
+ *
+ * After the various failure cases, the cache lock is proactively removed, so
+ * that another request is given the opportunity to attempt to cache without
+ * waiting for a potentially slow client to acknowledge the failure.
  */
 
 static int cache_save_filter(ap_filter_t *f, apr_bucket_brigade *in)
@@ -328,6 +348,7 @@
     cache_info *info = NULL;
     char *reason;
     apr_pool_t *p;
+    apr_bucket *e;
 
     conf = (cache_server_conf *) ap_get_module_config(r->server->module_config,
                                                       &cache_module);
@@ -370,7 +391,19 @@
             ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
                          "cache: Cache provider's store_body failed!");
             ap_remove_output_filter(f);
+
+            /* give someone else the chance to cache the file */
+            ap_cache_remove_lock(conf, r, cache->handle ?
+                    (char *)cache->handle->cache_obj->key : NULL, NULL);
         }
+        else {
+
+        	/* proactively remove the lock as soon as we see the eos bucket */
+            ap_cache_remove_lock(conf, r, cache->handle ?
+                    (char *)cache->handle->cache_obj->key : NULL, in);
+
+        }
+
         return ap_pass_brigade(f->next, in);
     }
 
@@ -441,17 +474,17 @@
          * telling us to serve the cached copy.
          */
         if (exps != NULL || cc_out != NULL) {
-            /* We are also allowed to cache any response given that it has a 
-             * valid Expires or Cache Control header. If we find a either of 
-             * those here,  we pass request through the rest of the tests. From 
+            /* We are also allowed to cache any response given that it has a
+             * valid Expires or Cache Control header. If we find a either of
+             * those here,  we pass request through the rest of the tests. From
              * the RFC:
              *
-             * A response received with any other status code (e.g. status 
-             * codes 302 and 307) MUST NOT be returned in a reply to a 
-             * subsequent request unless there are cache-control directives or 
-             * another header(s) that explicitly allow it. For example, these 
-             * include the following: an Expires header (section 14.21); a 
-             * "max-age", "s-maxage",  "must-revalidate", "proxy-revalidate", 
+             * A response received with any other status code (e.g. status
+             * codes 302 and 307) MUST NOT be returned in a reply to a
+             * subsequent request unless there are cache-control directives or
+             * another header(s) that explicitly allow it. For example, these
+             * include the following: an Expires header (section 14.21); a
+             * "max-age", "s-maxage",  "must-revalidate", "proxy-revalidate",
              * "public" or "private" cache-control directive (section 14.9).
              */
         }
@@ -544,7 +577,7 @@
                               "*", NULL)) {
         reason = "Vary header contains '*'";
     }
-    else if (apr_table_get(r->subprocess_env, "no-cache") != NULL) { 
+    else if (apr_table_get(r->subprocess_env, "no-cache") != NULL) {
         reason = "environment variable 'no-cache' is set";
     }
     else if (r->no_cache) {
@@ -560,6 +593,10 @@
         /* remove this filter from the chain */
         ap_remove_output_filter(f);
 
+        /* remove the lock file unconditionally */
+        ap_cache_remove_lock(conf, r, cache->handle ?
+                (char *)cache->handle->cache_obj->key : NULL, NULL);
+
         /* ship the data up the stack */
         return ap_pass_brigade(f->next, in);
     }
@@ -584,7 +621,6 @@
         /* if we don't get the content-length, see if we have all the
          * buckets and use their length to calculate the size
          */
-        apr_bucket *e;
         int all_buckets_here=0;
         int unresolved_length = 0;
         size=0;
@@ -661,6 +697,8 @@
     if (rv != OK) {
         /* Caching layer declined the opportunity to cache the response */
         ap_remove_output_filter(f);
+        ap_cache_remove_lock(conf, r, cache->handle ?
+                (char *)cache->handle->cache_obj->key : NULL, NULL);
         return ap_pass_brigade(f->next, in);
     }
 
@@ -849,16 +887,23 @@
                 ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
                      "cache: attempt to remove url from cache unsuccessful.");
             }
+
         }
 
+        /* let someone else attempt to cache */
+        ap_cache_remove_lock(conf, r, cache->handle ?
+                (char *)cache->handle->cache_obj->key : NULL, NULL);
+
         return ap_pass_brigade(f->next, bb);
     }
 
-    if(rv != APR_SUCCESS) {
+    if (rv != APR_SUCCESS) {
         ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
                      "cache: store_headers failed");
-        ap_remove_output_filter(f);
 
+        ap_remove_output_filter(f);
+        ap_cache_remove_lock(conf, r, cache->handle ?
+                (char *)cache->handle->cache_obj->key : NULL, NULL);
         return ap_pass_brigade(f->next, in);
     }
 
@@ -867,8 +912,15 @@
         ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
                      "cache: store_body failed");
         ap_remove_output_filter(f);
+        ap_cache_remove_lock(conf, r, cache->handle ?
+                (char *)cache->handle->cache_obj->key : NULL, NULL);
+        return ap_pass_brigade(f->next, in);
     }
 
+    /* proactively remove the lock as soon as we see the eos bucket */
+    ap_cache_remove_lock(conf, r, cache->handle ?
+            (char *)cache->handle->cache_obj->key : NULL, in);
+
     return ap_pass_brigade(f->next, in);
 }
 
@@ -912,6 +964,7 @@
         ap_remove_output_filter(f);
         return ap_pass_brigade(f->next, in);
     }
+
     /* Now remove this cache entry from the cache */
     cache_remove_url(cache, r->pool);
 
@@ -925,6 +978,7 @@
 
 static void * create_cache_config(apr_pool_t *p, server_rec *s)
 {
+    const char *tmppath;
     cache_server_conf *ps = apr_pcalloc(p, sizeof(cache_server_conf));
 
     /* array of URL prefixes for which caching is enabled */
@@ -957,6 +1011,13 @@
     /* array of identifiers that should not be used for key calculation */
     ps->ignore_session_id = apr_array_make(p, 10, sizeof(char *));
     ps->ignore_session_id_set = CACHE_IGNORE_SESSION_ID_UNSET;
+    ps->lock = 0; /* thundering herd lock defaults to off */
+    ps->lock_set = 0;
+    apr_temp_dir_get(&tmppath, p);
+    if (tmppath) {
+        ps->lockpath = apr_pstrcat(p, tmppath, DEFAULT_CACHE_LOCKPATH, NULL);
+    }
+    ps->lockmaxage = apr_time_from_sec(DEFAULT_CACHE_MAXAGE);
     return ps;
 }
 
@@ -1010,6 +1071,18 @@
         (overrides->ignore_session_id_set == CACHE_IGNORE_SESSION_ID_UNSET)
         ? base->ignore_session_id
         : overrides->ignore_session_id;
+    ps->lock =
+        (overrides->lock_set == 0)
+        ? base->lock
+        : overrides->lock;
+    ps->lockpath =
+        (overrides->lockpath_set == 0)
+        ? base->lockpath
+        : overrides->lockpath;
+    ps->lockmaxage =
+        (overrides->lockmaxage_set == 0)
+        ? base->lockmaxage
+        : overrides->lockmaxage;
     return ps;
 }
 static const char *set_cache_ignore_no_last_mod(cmd_parms *parms, void *dummy,
@@ -1229,6 +1302,55 @@
     return NULL;
 }
 
+static const char *set_cache_lock(cmd_parms *parms, void *dummy,
+                                                int flag)
+{
+    cache_server_conf *conf;
+
+    conf =
+        (cache_server_conf *)ap_get_module_config(parms->server->module_config,
+                                                  &cache_module);
+    conf->lock = flag;
+    conf->lock_set = 1;
+    return NULL;
+}
+
+static const char *set_cache_lock_path(cmd_parms *parms, void *dummy,
+                                    const char *arg)
+{
+    cache_server_conf *conf;
+
+    conf =
+        (cache_server_conf *)ap_get_module_config(parms->server->module_config,
+                                                  &cache_module);
+
+    conf->lockpath = ap_server_root_relative(parms->pool, arg);
+    if (!conf->lockpath) {
+        return apr_pstrcat(parms->pool, "Invalid CacheLockPath path ",
+                           arg, NULL);
+    }
+    conf->lockpath_set = 1;
+    return NULL;
+}
+
+static const char *set_cache_lock_maxage(cmd_parms *parms, void *dummy,
+                                    const char *arg)
+{
+    cache_server_conf *conf;
+    apr_int64_t seconds;
+
+    conf =
+        (cache_server_conf *)ap_get_module_config(parms->server->module_config,
+                                                  &cache_module);
+    seconds = apr_atoi64(arg);
+    if (seconds <= 0) {
+        return "CacheLockMaxAge value must be a non-zero positive integer";
+    }
+    conf->lockmaxage = apr_time_from_sec(seconds);
+    conf->lockmaxage_set = 1;
+    return NULL;
+}
+
 static int cache_post_config(apr_pool_t *p, apr_pool_t *plog,
                              apr_pool_t *ptemp, server_rec *s)
 {
@@ -1288,6 +1410,15 @@
     AP_INIT_TAKE1("CacheLastModifiedFactor", set_cache_factor, NULL, RSRC_CONF,
                   "The factor used to estimate Expires date from "
                   "LastModified date"),
+    AP_INIT_FLAG("CacheLock", set_cache_lock,
+                 NULL, RSRC_CONF,
+                 "Enable or disable the thundering herd lock."),
+    AP_INIT_TAKE1("CacheLockPath", set_cache_lock_path, NULL, RSRC_CONF,
+                  "The thundering herd lock path. Defaults to the '"
+                  DEFAULT_CACHE_LOCKPATH "' directory in the system "
+                  "temp directory."),
+    AP_INIT_TAKE1("CacheLockMaxAge", set_cache_lock_maxage, NULL, RSRC_CONF,
+                  "Maximum age of any thundering herd lock."),
     {NULL}
 };
 

Modified: httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h Sat Feb 27 18:54:40 2010
@@ -24,7 +24,7 @@
  */
 
 #ifndef MOD_CACHE_H
-#define MOD_CACHE_H 
+#define MOD_CACHE_H
 
 #define CORE_PRIVATE
 
@@ -86,9 +86,13 @@
 #define DEFAULT_CACHE_MAXEXPIRE MSEC_ONE_DAY
 #define DEFAULT_CACHE_EXPIRE    MSEC_ONE_HR
 #define DEFAULT_CACHE_LMFACTOR  (0.1)
+#define DEFAULT_CACHE_MAXAGE    5
+#define DEFAULT_CACHE_LOCKPATH "/mod_cache-lock"
+#define CACHE_LOCKNAME_KEY "mod_cache-lockname"
+#define CACHE_LOCKFILE_KEY "mod_cache-lockfile"
 
-/* Create a set of PROXY_DECLARE(type), PROXY_DECLARE_NONSTD(type) and 
- * PROXY_DECLARE_DATA with appropriate export and import tags for the platform
+/* Create a set of CACHE_DECLARE(type), CACHE_DECLARE_NONSTD(type) and
+ * CACHE_DECLARE_DATA with appropriate export and import tags for the platform
  */
 #if !defined(WIN32)
 #define CACHE_DECLARE(type)            type
@@ -134,7 +138,7 @@
     int factor_set;
     /** ignore the last-modified header when deciding to cache this request */
     int no_last_mod_ignore_set;
-    int no_last_mod_ignore; 
+    int no_last_mod_ignore;
     /** ignore client's requests for uncached responses */
     int ignorecachecontrol;
     int ignorecachecontrol_set;
@@ -159,6 +163,13 @@
     #define CACHE_IGNORE_SESSION_ID_SET   1
     #define CACHE_IGNORE_SESSION_ID_UNSET 0
     int ignore_session_id_set;
+    /* thundering herd lock */
+    int lock;
+    int lock_set;
+    const char *lockpath;
+    int lockpath_set;
+    apr_time_t lockmaxage;
+    int lockmaxage_set;
 } cache_server_conf;
 
 /* cache info information */
@@ -173,11 +184,11 @@
 
 /* cache handle information */
 
-/* XXX TODO On the next structure change/MMN bump, 
+/* XXX TODO On the next structure change/MMN bump,
  * count must become an apr_off_t, representing
  * the potential size of disk cached objects.
  * Then dig for
- * "XXX Bad Temporary Cast - see cache_object_t notes" 
+ * "XXX Bad Temporary Cast - see cache_object_t notes"
  */
 typedef struct cache_object cache_object_t;
 struct cache_object {
@@ -206,7 +217,7 @@
     apr_status_t (*store_headers)(cache_handle_t *h, request_rec *r, cache_info *i);
     apr_status_t (*store_body)(cache_handle_t *h, request_rec *r, apr_bucket_brigade *b);
     apr_status_t (*recall_headers) (cache_handle_t *h, request_rec *r);
-    apr_status_t (*recall_body) (cache_handle_t *h, apr_pool_t *p, apr_bucket_brigade *bb);

+    apr_status_t (*recall_body) (cache_handle_t *h, apr_pool_t *p, apr_bucket_brigade *bb);
     int (*create_entity) (cache_handle_t *h, request_rec *r,
                            const char *urlkey, apr_off_t len);
     int (*open_entity) (cache_handle_t *h, request_rec *r,
@@ -260,6 +271,45 @@
 CACHE_DECLARE(int) ap_cache_check_freshness(cache_handle_t *h, request_rec *r);
 
 /**
+ * Try obtain a cache wide lock on the given cache key.
+ *
+ * If we return APR_SUCCESS, we obtained the lock, and we are clear to
+ * proceed to the backend. If we return APR_EEXISTS, the the lock is
+ * already locked, someone else has gone to refresh the backend data
+ * already, so we must return stale data with a warning in the mean
+ * time. If we return anything else, then something has gone pear
+ * shaped, and we allow the request through to the backend regardless.
+ *
+ * This lock is created from the request pool, meaning that should
+ * something go wrong and the lock isn't deleted on return of the
+ * request headers from the backend for whatever reason, at worst the
+ * lock will be cleaned up when the request is dies or finishes.
+ *
+ * If something goes truly bananas and the lock isn't deleted when the
+ * request dies, the lock will be trashed when its max-age is reached,
+ * or when a request arrives containing a Cache-Control: no-cache. At
+ * no point is it possible for this lock to permanently deny access to
+ * the backend.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_try_lock(cache_server_conf *conf,
+		request_rec *r, char *key);
+
+/**
+ * Remove the cache lock, if present.
+ *
+ * First, try to close the file handle, whose delete-on-close should
+ * kill the file. Otherwise, just delete the file by name.
+ *
+ * If no lock name has yet been calculated, do the calculation of the
+ * lock name first before trying to delete the file.
+ *
+ * If an optional bucket brigade is passed, the lock will only be
+ * removed if the bucket brigade contains an EOS bucket.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_remove_lock(cache_server_conf *conf,
+		request_rec *r, char *key, apr_bucket_brigade *bb);
+
+/**
  * Merge in cached headers into the response
  * @param h cache_handle_t
  * @param r request_rec
@@ -330,8 +380,8 @@
 #define CACHE_DECLARE_DATA             __declspec(dllimport)
 #endif
 
-APR_DECLARE_OPTIONAL_FN(apr_status_t, 
-                        ap_cache_generate_key, 
+APR_DECLARE_OPTIONAL_FN(apr_status_t,
+                        ap_cache_generate_key,
                         (request_rec *r, apr_pool_t*p, char**key ));
 
 



Mime
View raw message