jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Al Forbes <forbes...@googlemail.com>
Subject Re: index cache corruption
Date Wed, 12 Aug 2009 15:33:54 GMT
Excellent news...that's a pretty hard bug to spot.

Many thanks for your effort.


2009/8/12 Aaron Smuts <asmuts@yahoo.com>

> I think I found it.  During partial key removeall the Disk Element
> Descriptor, which identifies the slot on disk, was being added to the
> recylebin.  Then the removeSingleItem was called for each match.  The remove
> single item method also adds the item to the recylebin.  The recyclebin is
> not a set.  The spot would get entered twice.  This is very bad.  Here's
> why:
>
> The spot gets used.  Another put uses the same spot.  Both keys in the key
> map point to the same spot.  A get for the first key will pull the data for
> the second.  Not so good.
>
> I'm verifying this and I'll have a fix soon.
>
> Aaron
>
> --- On Wed, 8/12/09, Aaron Smuts <asmuts@yahoo.com> wrote:
>
> > From: Aaron Smuts <asmuts@yahoo.com>
> > Subject: RE: index cache corruption
> > To: "JCS Users List" <jcs-users@jakarta.apache.org>
> > Date: Wednesday, August 12, 2009, 6:52 AM
> >
> > So were you getting back the wrong data or data that should
> > have been deleted?
> >
> > Are you running a recent temp build?
> >
> > I'll dig into the partial key removal.  This should
> > probably deprecated for the remove matching method. . .
> > .  Either way, it should work.
> >
> > Aaron
> >
> > --- On Tue, 8/11/09, Tim Cronin <Tim.Cronin@autonomy.com>
> > wrote:
> >
> > > From: Tim Cronin <Tim.Cronin@autonomy.com>
> > > Subject: RE: index cache corruption
> > > To: "JCS Users List" <jcs-users@jakarta.apache.org>
> > > Date: Tuesday, August 11, 2009, 2:30 PM
> > > Good to know, thanks.
> > >
> > > Yes that's when we see it too, under high load.
> > >
> > > For us it seems to happen around removes, we use the
> > key:
> > > linkage and
> > > It happens to data that's linked and would get cleared
> > with
> > > the key:
> > > call.
> > >
> > >
> > > -----Original Message-----
> > > From: Al Forbes [mailto:forbes.al@googlemail.com]
> > >
> > > Sent: Tuesday, August 11, 2009 4:13 PM
> > > To: JCS Users List
> > > Subject: Re: index cache corruption
> > >
> > > Hi Tim,
> > >
> > > I had the same problem. It seems to happen under high
> > load,
> > > with fairly
> > > large objects, but I could not reproduce it offline.
> > I
> > > changed to using
> > > a
> > > JDBC cache and have not had a problem since then.
> > >
> > > We still use disk caches for more static data - mostly
> > read
> > > only, and
> > > the
> > > problem never happens there, so I assume it's the
> > writes
> > > that cause the
> > > corruption.
> > >
> > > Regards
> > > Al
> > >
> > > 2009/8/11 Tim Cronin <Tim.Cronin@autonomy.com>
> > >
> > > > We initially were using indexed disk cache but
> > ran
> > > into cache
> > > corruption
> > > > where the data returned for a key would not be
> > the
> > > data associated for
> > > > that key. I haven't been able to come up with a
> > good
> > > repro scenario...
> > > >
> > > > We had to work around it with the following
> > code:
> > > >
> > > >  /**
> > > >   * is the cache element not the
> > > correct object for the key
> > > >   * @param key the current key
> > > >   * @param element the current element
> > > associated with the key
> > > >   * @param hasReadLock whether caller
> > > has read lock
> > > >   * @return true if key mismatch els
> > > false
> > > >   * @throws InterruptedException if
> > > locking fails
> > > >   */
> > > >  private boolean isCorrupt(String key,
> > > ICacheElement element, boolean
> > > > hasReadLock) throws InterruptedException
> > > >  {
> > > >    boolean corrupt =
> > > !key.equals(element.getKey());
> > > >    if (corrupt)
> > > >    {
> > > >      mLogger.error("cache corruption!!!
> > > [" + (mCorruptionCounter++) +
> > > > "]");
> > > >
> > > >      if (mLogger.isDebugEnabled())
> > > >      {
> > > >       mLogger.error("culprit
> > > stack...", new Exception("cache
> > > > corruption"));
> > > >      }
> > > >
> > > >      try
> > > >      {
> > > >        if (hasReadLock)
> > > >        {
> > > >
> > > mLock.readLock().release();
> > > >        }
> > > >
> > > mLock.writeLock().acquire();
> > > >        try
> > > >        {
> > > >
> > > mLogger.error("purging " + key);
> > > >          mCache.remove(key);
> > > >          mCache.remove(key +
> > > ":");
> > > >        }
> > > >        catch (Exception e)
> > > >        {
> > > >
> > > mLogger.error("failed to purge key " + key, e);
> > > >        }
> > > >        try
> > > >        {
> > > >          String k =
> > > (String)element.getKey();
> > > >
> > > mLogger.error("purging " + k);
> > > >          mCache.remove(k);
> > > >          mCache.remove(k +
> > > ":");
> > > >        }
> > > >        catch (Exception e)
> > > >        {
> > > >
> > > mLogger.error("failed to purge key " +
> > element.getKey(),
> > > e);
> > > >        }
> > > >      }
> > > >      finally
> > > >      {
> > > >        if (hasReadLock)
> > > >        {
> > > >
> > > mLock.readLock().acquire();
> > > >        }
> > > >
> > > mLock.writeLock().release();
> > > >      }
> > > >    }
> > > >    return corrupt;
> > > >  }
> > > >
> > > >
> > > >
> > >
> > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail: jcs-users-help@jakarta.apache.org
> > > >
> > > >
> > >
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: jcs-users-help@jakarta.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: jcs-users-help@jakarta.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jcs-users-help@jakarta.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message