cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geoff Howard" <coc...@leverageweb.com>
Subject External/Event Based Cache Invalidation (somewhat long)
Date Sat, 21 Jun 2003 14:30:49 GMT
Below is the larger picture I envision for a new kind of cache
invaliditation
that I've needed in the past and comes up in requests from people using EJB
or
database driven data that is cacheable.  I'd love feedback from anyone who's
interested.  I've BCC'd a few people who I've talked with about this in the
past.

----------------------------------------------------------------------------

I've committed some little things into the cocoon scratchpad - part of some
experimental work for external cache invalidation, which as I've thought
about it is really "event" based cache invalidation.

Why? Because the events need not be external to Cocoon, and when you think
about it, all cache validities are "external" in that the Cache validity
object needs to consult some external reference - the filesystem, the system
time, etc.

But event based invalidation has some differences in the way they need to be
handled.  The other validities can reasonably be expected to check with
their
external resources (time, filesystem, etc) when retrieved from cache and
isValid() is called.  But with events, the subsystem which received the
events
would need to store them all up waiting for the call to isValid() which
depending on other factors might never come.  It seems to me more fitting
with the transient nature of events to act on them when they arrive and then
discard them.

That leaves only several choices
- make the event know about what cache keys it needs to remove.  This is
the only solution currently available and it has in practice meant hard
coding the event-key relationship somewhere and manually maintaining it.
Not good IMHO.
- search through every cached item on the receipt of an event to see if
it is now invalid in light of the current event.  Also not good.
- Extend the cache to provide support for the cache-event concept.  This is
the tack I'm taking.  Essentially, this solution involves the CacheImpl
keeping track of mapping the Event's declared in an EventValidity to the
key under which that response is stored in the cache.

The "glue" that is missing is the org.apache.cocoon.caching.impl.CacheImpl
extension, because it won't compile without the change I made to
sourceresolve,
which is not yet in cocoon's cvs.  For some odd reason I'm having a hard
time
building the sourceresolve package using its build script.  It's also not
"done" as noted below - but I'd love others to be able to work on it.

Here are the issues the "event aware" CacheImpl would need to take care of:
- during store() it gets the SourceValidity[] from the CachedResponse and
looks for instanceof EventValidity (recursively for AggregatedValidity).
- if found, it calls EventValidity.getEvent() and stores the key-event
mapping.
- expose a removeByEvent(Event e) method that can be called by the specific
event-handling component.  This could be a jms listener (as I've orginally
envisioned it) or an http/soap based system (as in the ESI patch that was in
bugzilla) or even a cocoon Action or Flow, or some combination of all of the
above.
- When the key is ejected from the cache for other reasons (another pipeline
component signalled invalid for example) I think it's necessary to at that
moment remove the event-key mapping entry.  This introduces a complication
in the data structure used to store these mappings as I mention below.  I
also haven't looked into the effect of the store janitor - if it acts
directly
on the Store without going through the CacheImpl wrapper, that introduces a
wrinkle.

Most of the above is accounted for - except for the data structure
to store the event-key mappings.  As discussed above, it needs to:
- allow duplicate keys (each event may uncache multiple pipelines, and each
pipeline might be uncached by any of multiple events).  So it needs a Bag.
- allow lookup of mappings based on either event or key.  Jakarta Commons
Collections has a DoubleOrderedMap, but not a DoubleOrderedBag.  Bummer.
- be persistent across shutdown and startup, and needs to recover gracefully
when the cache is missing (like when it's been manually deleted)
- be efficient

I have made an assumption so far that I'd like tested by some sharp minds.
When a removeByEvent() is received, the CacheImpl would do something like
PipelineCacheKey[] getByEvent() on its datastructure.  This would rely on
hashCode() and equals() of Event() to locate relevant events.  I think this
works well for true "equals" type of information: like "table_name" and
"primary_key" -- if they are both equal, the event has happened.  But there
may be some where a "greater than" or "less than" or worse yet, a kind of
wild card lookup might need to be supported.  Can that be accomodated by a
Collections sort of implementation, or does something more flexible need
to be invented? As it stands, you might implement hashCode() in a
way that will cause intentional collisions and rely on equals() to sort
things out.  Is that crazy?

Geoff

> -----Original Message-----
> From: ghoward@apache.org [mailto:ghoward@apache.org]
> Sent: Friday, June 20, 2003 11:36 PM
> To: cocoon-2.1-cvs@apache.org
> Subject: cvs commit:
> cocoon-2.1/src/scratchpad/src/org/apache/cocoon/caching/validity
> NameValueEvent.java Event.java EventValidity.java NamedEvent.java
>
>
> ghoward     2003/06/20 20:36:15
>
>   Added:       src/scratchpad/src/org/apache/cocoon/caching/validity
>                         NameValueEvent.java Event.java EventValidity.java
>                         NamedEvent.java
>   Log:
>   Experiment with external cache invalidation.  An EventAwareCacheImpl
>   can't be committed yet because it relies on the latest sourceresolve
>   cvs.
>
>   Revision  Changes    Path
>   1.1
> cocoon-2.1/src/scratchpad/src/org/apache/cocoon/caching/validity/N
> ameValueEvent.java
>
>   Index: NameValueEvent.java
>   ===================================================================
>   /*
>
>
> ==================================================================
> ==========
>                      The Apache Software License, Version 1.1
>
> ==================================================================
> ==========
>
>    Copyright (C) 1999-2003 The Apache Software Foundation. All
> rights reserved.
>
>    Redistribution and use in source and binary forms, with or
> without modifica-
>    tion, are permitted provided that the following conditions are met:
>
>    1. Redistributions of  source code must  retain the above
> copyright  notice,
>       this list of conditions and the following disclaimer.
>
>    2. Redistributions in binary form must reproduce the above
> copyright notice,
>       this list of conditions and the following disclaimer in the
> documentation
>       and/or other materials provided with the distribution.
>
>    3. The end-user documentation included with the
> redistribution, if any, must
>       include  the following  acknowledgment:  "This product
> includes  software
>       developed  by the  Apache Software Foundation
> (http://www.apache.org/)."
>       Alternately, this  acknowledgment may  appear in the
> software itself,  if
>       and wherever such third-party acknowledgments normally appear.
>
>    4. The names "Apache Cocoon" and  "Apache Software Foundation"
> must  not  be
>       used to  endorse or promote  products derived from  this
> software without
>       prior written permission. For written permission, please contact
>       apache@apache.org.
>
>    5. Products  derived from this software may not  be called
> "Apache", nor may
>       "Apache" appear  in their name,  without prior written
> permission  of the
>       Apache Software Foundation.
>
>    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR
> IMPLIED WARRANTIES,
>    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
> MERCHANTABILITY AND
>    FITNESS  FOR A PARTICULAR  PURPOSE ARE  DISCLAIMED.  IN NO
> EVENT SHALL  THE
>    APACHE SOFTWARE  FOUNDATION  OR ITS CONTRIBUTORS  BE LIABLE
> FOR  ANY DIRECT,
>    INDIRECT, INCIDENTAL, SPECIAL,  EXEMPLARY, OR CONSEQUENTIAL
> DAMAGES (INCLU-
>    DING, BUT NOT LIMITED TO, PROCUREMENT  OF SUBSTITUTE GOODS OR
> SERVICES; LOSS
>    OF USE, DATA, OR  PROFITS; OR BUSINESS  INTERRUPTION)  HOWEVER
> CAUSED AND ON
>    ANY  THEORY OF LIABILITY,  WHETHER  IN CONTRACT,  STRICT
> LIABILITY,  OR TORT
>    (INCLUDING  NEGLIGENCE OR  OTHERWISE) ARISING IN  ANY WAY OUT
> OF THE  USE OF
>    THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>
>   */
>   package org.apache.cocoon.caching.validity;
>
>   /**
>    * Very experimental start at external cache invalidation.
>    * Warning - API very unstable.  Do not use!  In fact, if this
>    * becomes useful, it would probably move to Excalibur SourceResolve.
>    *
>    * An external uncache event that consists of a name/value pair.
>    * An example might be "table_name", "primary_key"
>    *
>    * @author Geoff Howard (ghoward@apache.org)
>    * @version $CVS$
>    */
>   public class NameValueEvent extends Event {
>
>       private String m_name;
>       private String m_value;
>
>   	public NameValueEvent(String name, String value) {
>           m_name = name;
>           m_value = value;
>   	}
>
>       private String getName() {
>           return m_name;
>       }
>
>       private String getValue() {
>           return m_value;
>       }
>
>   	public boolean equals(Event e) {
>   		if (e instanceof NameValueEvent) {
>               NameValueEvent nve = (NameValueEvent)e;
>               return ( m_name.equals(nve.getName()) &&
>                   m_value.equals(nve.getValue()) );
>   		}
>   		return false;
>   	}
>
>   }
>
>
>
>   1.1
> cocoon-2.1/src/scratchpad/src/org/apache/cocoon/caching/validity/E
> vent.java
>
>   Index: Event.java
>   ===================================================================
>   /*
>
>
> ==================================================================
> ==========
>                      The Apache Software License, Version 1.1
>
> ==================================================================
> ==========
>
>    Copyright (C) 1999-2003 The Apache Software Foundation. All
> rights reserved.
>
>    Redistribution and use in source and binary forms, with or
> without modifica-
>    tion, are permitted provided that the following conditions are met:
>
>    1. Redistributions of  source code must  retain the above
> copyright  notice,
>       this list of conditions and the following disclaimer.
>
>    2. Redistributions in binary form must reproduce the above
> copyright notice,
>       this list of conditions and the following disclaimer in the
> documentation
>       and/or other materials provided with the distribution.
>
>    3. The end-user documentation included with the
> redistribution, if any, must
>       include  the following  acknowledgment:  "This product
> includes  software
>       developed  by the  Apache Software Foundation
> (http://www.apache.org/)."
>       Alternately, this  acknowledgment may  appear in the
> software itself,  if
>       and wherever such third-party acknowledgments normally appear.
>
>    4. The names "Apache Cocoon" and  "Apache Software Foundation"
> must  not  be
>       used to  endorse or promote  products derived from  this
> software without
>       prior written permission. For written permission, please contact
>       apache@apache.org.
>
>    5. Products  derived from this software may not  be called
> "Apache", nor may
>       "Apache" appear  in their name,  without prior written
> permission  of the
>       Apache Software Foundation.
>
>    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR
> IMPLIED WARRANTIES,
>    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
> MERCHANTABILITY AND
>    FITNESS  FOR A PARTICULAR  PURPOSE ARE  DISCLAIMED.  IN NO
> EVENT SHALL  THE
>    APACHE SOFTWARE  FOUNDATION  OR ITS CONTRIBUTORS  BE LIABLE
> FOR  ANY DIRECT,
>    INDIRECT, INCIDENTAL, SPECIAL,  EXEMPLARY, OR CONSEQUENTIAL
> DAMAGES (INCLU-
>    DING, BUT NOT LIMITED TO, PROCUREMENT  OF SUBSTITUTE GOODS OR
> SERVICES; LOSS
>    OF USE, DATA, OR  PROFITS; OR BUSINESS  INTERRUPTION)  HOWEVER
> CAUSED AND ON
>    ANY  THEORY OF LIABILITY,  WHETHER  IN CONTRACT,  STRICT
> LIABILITY,  OR TORT
>    (INCLUDING  NEGLIGENCE OR  OTHERWISE) ARISING IN  ANY WAY OUT
> OF THE  USE OF
>    THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>
>   */
>   package org.apache.cocoon.caching.validity;
>
>   /**
>    * Very experimental start at external cache invalidation.
>    * Warning - API very unstable.  Do not use!  In fact, if this
>    * becomes useful, it would probably move to Excalibur SourceResolve.
>    *
>    * Base class encapsulating the information about an external
>    * uncache event.
>    *
>    * @author Geoff Howard (ghoward@apache.org)
>    *
>    * @version $CVS$
>    */
>   public abstract class Event {
>
>       public abstract boolean equals(Event e);
>
>       public boolean equals(Object o) {
>           if (o instanceof Event) {
>               return equals((Event)o);
>           }
>           return false;
>       }
>
>   }
>
>
>
>   1.1
> cocoon-2.1/src/scratchpad/src/org/apache/cocoon/caching/validity/E
> ventValidity.java
>
>   Index: EventValidity.java
>   ===================================================================
>   /*
>
>
> ==================================================================
> ==========
>                      The Apache Software License, Version 1.1
>
> ==================================================================
> ==========
>
>    Copyright (C) 1999-2003 The Apache Software Foundation. All
> rights reserved.
>
>    Redistribution and use in source and binary forms, with or
> without modifica-
>    tion, are permitted provided that the following conditions are met:
>
>    1. Redistributions of  source code must  retain the above
> copyright  notice,
>       this list of conditions and the following disclaimer.
>
>    2. Redistributions in binary form must reproduce the above
> copyright notice,
>       this list of conditions and the following disclaimer in the
> documentation
>       and/or other materials provided with the distribution.
>
>    3. The end-user documentation included with the
> redistribution, if any, must
>       include  the following  acknowledgment:  "This product
> includes  software
>       developed  by the  Apache Software Foundation
> (http://www.apache.org/)."
>       Alternately, this  acknowledgment may  appear in the
> software itself,  if
>       and wherever such third-party acknowledgments normally appear.
>
>    4. The names "Apache Cocoon" and  "Apache Software Foundation"
> must  not  be
>       used to  endorse or promote  products derived from  this
> software without
>       prior written permission. For written permission, please contact
>       apache@apache.org.
>
>    5. Products  derived from this software may not  be called
> "Apache", nor may
>       "Apache" appear  in their name,  without prior written
> permission  of the
>       Apache Software Foundation.
>
>    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR
> IMPLIED WARRANTIES,
>    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
> MERCHANTABILITY AND
>    FITNESS  FOR A PARTICULAR  PURPOSE ARE  DISCLAIMED.  IN NO
> EVENT SHALL  THE
>    APACHE SOFTWARE  FOUNDATION  OR ITS CONTRIBUTORS  BE LIABLE
> FOR  ANY DIRECT,
>    INDIRECT, INCIDENTAL, SPECIAL,  EXEMPLARY, OR CONSEQUENTIAL
> DAMAGES (INCLU-
>    DING, BUT NOT LIMITED TO, PROCUREMENT  OF SUBSTITUTE GOODS OR
> SERVICES; LOSS
>    OF USE, DATA, OR  PROFITS; OR BUSINESS  INTERRUPTION)  HOWEVER
> CAUSED AND ON
>    ANY  THEORY OF LIABILITY,  WHETHER  IN CONTRACT,  STRICT
> LIABILITY,  OR TORT
>    (INCLUDING  NEGLIGENCE OR  OTHERWISE) ARISING IN  ANY WAY OUT
> OF THE  USE OF
>    THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>
>   */
>   package org.apache.cocoon.caching.validity;
>
>   import org.apache.excalibur.source.SourceValidity;
>
>   /**
>    * Very experimental start at external cache invalidation.
>    * Warning - API very unstable.  Do not use!  In fact, if this
>    * becomes useful, it would probably move to Excalibur SourceResolve.
>    *
>    * The SourceValidity object for cache invalidation based on
>    * external events.
>    *
>    * @author Geoff Howard(ghoward@apache.org)
>    * @version $CVS$
>    */
>   public class EventValidity implements SourceValidity {
>
>       private Event m_event;
>
>       public EventValidity(Event ev) {
>           m_event = ev;
>       }
>
>       public Event getEvent() {
>           return m_event;
>       }
>
>   	/** Basic implementation is always valid until event signals
>        *  otherwise.  May never need other behavior.
>   	 */
>   	public int isValid() {
>   		return VALID;
>   	}
>
>   	public int isValid(SourceValidity sv) {
>   		if (sv instanceof EventValidity) {
>               return VALID;
>   		}
>           return INVALID;
>   	}
>   }
>
>
>
>   1.1
> cocoon-2.1/src/scratchpad/src/org/apache/cocoon/caching/validity/N
> amedEvent.java
>
>   Index: NamedEvent.java
>   ===================================================================
>   /*
>
>
> ==================================================================
> ==========
>                      The Apache Software License, Version 1.1
>
> ==================================================================
> ==========
>
>    Copyright (C) 1999-2003 The Apache Software Foundation. All
> rights reserved.
>
>    Redistribution and use in source and binary forms, with or
> without modifica-
>    tion, are permitted provided that the following conditions are met:
>
>    1. Redistributions of  source code must  retain the above
> copyright  notice,
>       this list of conditions and the following disclaimer.
>
>    2. Redistributions in binary form must reproduce the above
> copyright notice,
>       this list of conditions and the following disclaimer in the
> documentation
>       and/or other materials provided with the distribution.
>
>    3. The end-user documentation included with the
> redistribution, if any, must
>       include  the following  acknowledgment:  "This product
> includes  software
>       developed  by the  Apache Software Foundation
> (http://www.apache.org/)."
>       Alternately, this  acknowledgment may  appear in the
> software itself,  if
>       and wherever such third-party acknowledgments normally appear.
>
>    4. The names "Apache Cocoon" and  "Apache Software Foundation"
> must  not  be
>       used to  endorse or promote  products derived from  this
> software without
>       prior written permission. For written permission, please contact
>       apache@apache.org.
>
>    5. Products  derived from this software may not  be called
> "Apache", nor may
>       "Apache" appear  in their name,  without prior written
> permission  of the
>       Apache Software Foundation.
>
>    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR
> IMPLIED WARRANTIES,
>    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
> MERCHANTABILITY AND
>    FITNESS  FOR A PARTICULAR  PURPOSE ARE  DISCLAIMED.  IN NO
> EVENT SHALL  THE
>    APACHE SOFTWARE  FOUNDATION  OR ITS CONTRIBUTORS  BE LIABLE
> FOR  ANY DIRECT,
>    INDIRECT, INCIDENTAL, SPECIAL,  EXEMPLARY, OR CONSEQUENTIAL
> DAMAGES (INCLU-
>    DING, BUT NOT LIMITED TO, PROCUREMENT  OF SUBSTITUTE GOODS OR
> SERVICES; LOSS
>    OF USE, DATA, OR  PROFITS; OR BUSINESS  INTERRUPTION)  HOWEVER
> CAUSED AND ON
>    ANY  THEORY OF LIABILITY,  WHETHER  IN CONTRACT,  STRICT
> LIABILITY,  OR TORT
>    (INCLUDING  NEGLIGENCE OR  OTHERWISE) ARISING IN  ANY WAY OUT
> OF THE  USE OF
>    THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>
>   */
>   package org.apache.cocoon.caching.validity;
>
>   /**
>    * Very experimental start at external cache invalidation.
>    * Warning - API very unstable.  Do not use!  In fact, if this
>    * becomes useful, it would probably move to Excalibur SourceResolve.
>    *
>    * An External cache event that consists of just a name.  Examples
>    * (not necessarily useful) could include "Easter" or "Shutdown"
>    *
>    * @author Geoff Howard (ghoward@apache.org)
>    * @version $CVS$
>    */
>   public class NamedEvent extends Event {
>
>       private String m_name;
>
>       public NamedEvent(String name) {
>           m_name = name;
>       }
>
>       private String getName() {
>           return m_name;
>       }
>
>   	public boolean equals(Event e) {
>           if (e instanceof NamedEvent) {
>               return m_name.equals(((NamedEvent)e).getName());
>           }
>   		return false;
>   	}
>
>
>   }
>
>
>
>
>


Mime
View raw message