From dev-return-61468-apmail-httpd-dev-archive=httpd.apache.org@httpd.apache.org Fri May 30 12:00:07 2008 Return-Path: Delivered-To: apmail-httpd-dev-archive@www.apache.org Received: (qmail 10676 invoked from network); 30 May 2008 12:00:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 30 May 2008 12:00:07 -0000 Received: (qmail 20775 invoked by uid 500); 30 May 2008 12:00:06 -0000 Delivered-To: apmail-httpd-dev-archive@httpd.apache.org Received: (qmail 20710 invoked by uid 500); 30 May 2008 12:00:06 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 20699 invoked by uid 99); 30 May 2008 12:00:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2008 05:00:06 -0700 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [64.236.221.40] (HELO atmail5.turner.com) (64.236.221.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2008 11:59:19 +0000 X-IronPort-AV: E=Sophos;i="4.27,566,1204520400"; d="scan'208";a="384634028" Received: from atlbh01.turner.com ([10.188.157.231]) by atliport02.turner.com with ESMTP; 30 May 2008 07:59:35 -0400 Received: from ATLMSG13.turner.com ([10.188.170.53]) by ATLBH01.turner.com with Microsoft SMTPSVC(6.0.3790.211); Fri, 30 May 2008 07:59:35 -0400 Received: from 10.188.174.6 ([10.188.174.6]) by ATLMSG13.turner.com ([10.188.170.53]) via Exchange Front-End Server atlwebaccess.turner.com ([10.188.157.196]) with Microsoft Exchange Server HTTP-DAV ; Fri, 30 May 2008 11:59:35 +0000 User-Agent: Microsoft-Entourage/12.10.0.080409 Date: Fri, 30 May 2008 07:59:32 -0400 Subject: Re: Adding purge/invalidation to mod_cache From: "Akins, Brian" To: "dev@httpd.apache.org" Message-ID: Thread-Topic: Adding purge/invalidation to mod_cache Thread-Index: AcjCRgdUdcgf8RE7TjSqf+aBe++eBAABpedt In-Reply-To: <483FE106.6040509@apache.org> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-OriginalArrivalTime: 30 May 2008 11:59:35.0162 (UTC) FILETIME=[A0D425A0:01C8C24C] X-Virus-Checked: Checked by ClamAV on apache.org How we handle purge: -Our cache is only disk based, so it is somewhat easier. -We have a hook that gets called after each cache store - ie, after we write a vary meta file, after we write a "real" meta file, after a datafile write (on close). -We also have a cache_get_info function that is exported (think curl_get_info) so that other modules can get info about cache objects (filenames, uri, sizes, etc.) -Our default "cache store" registers this post-store hook. It just writes the data out to a pipe - uri filename size expires\n -This piped program streams this info into a database (MySQL usually, as we had issues with SQLite) -Our cache manager is really just a fastcgi that uses the database to expire and purge cache objects. In a vary situation, we will get 3 post-store's for the initial cache: -first is the "main" key for "main" meta file -second is the varied key for the "real" meta file (the one that has headers, etc.) -third is the varied key/file for the data So a simple sql query for "select file from my_table where url='/my/url'" will return all 3. (LIKE and regex's work as well.) We also can easily keep track of cache size (select sum(object_size) from my_table), oldest entries, etc. Subsequent requests for same uri with different vary will add 2 entries to the database - varied meta and data. The fastcgi and pipe are because we prototyped all of this in perl and never rewrote it because it all works fairly well. Also, we had a few issues with mysql client directly in Apache and the pipe was just easier... -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies