cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Corin Moss" <Corin.M...@tvnz.co.nz>
Subject RE: Turning off default MRU store
Date Sat, 06 Mar 2004 01:48:21 GMT

Hiya,

That's a very pertinent set of questions - I'll answer as best I can
below, the only caveat is that I won't proclaim myself a JCS expert,
given that I was really only introduced to it a few days ago ;)

-----Original Message-----
From: Geoff Howard [mailto:cocoon@leverageweb.com]
Sent: Saturday, 6 March 2004 2:38 p.m.
To: dev@cocoon.apache.org
Subject: Re: Turning off default MRU store


Corin Moss wrote:

>Hi Guys,
>
>I think the thing to remember here is that it's the Disk Cache
>implementation that has this behaviour, and that it's very easy to use
>a different store type if the user feels it's necessary.  There are
>other, "safer" store implementations we can use - the only problem in
>this case is that the index is kept in memory (which is what makes it
>so fast.) If the user decides that this is inappropriate for them, the
>change to a different store type is very easy to make within the
>config.
>
>I think that moving to a filesystem store would be counter-productive,
>as the thing that JCS provides is total flexibility of store type.
>
>Just my thoughts :)
> 
>

Ok, I apologize that I just can't look into this myself right now, but
let me try to state what I perceive to be the things we really need so
we can narrow down whether this is the right direction to be going:

1)
- we need a high-performance in-memory cache
- we need that cache to be backed by some sort of persistence for two
reasons
  first, so that cached items aren't lost between server starts
  second so that items bumped off the memory cache can still be
preserved as the assumption is that pulling a cached item off disk is
still faster than regenerating the entire pipeline.  for example, the
most popular 100 items are served in memory, but the rest are on disk.
(and at shutdown, the 100 in-memory ones go to disk).

** is JCS (properly configured) a good way to meet both of these
requirements?

CM: Yes.  A JCS store can be configured to use an MRU store (the test
config provided does just this.) This MRU store is in turn configurable
- number of objects catered for, what type of store (memory etc.) The
swapping to disk based store is handled in the same way as Cocoon
currently does - objects are dropped out of MRU store into diskbased
store. The only difference is that the "disk based" store can in fact be
any of the several stores supported by JCS (remote server, disk, JISP
etc.)

At shutdown, things can be configured to store to disk (both store, and
index.)  The only issue is that in the event of an abnormal shutdown the
index can be lost (this is only when the index is kept in memory of
course - other store types which use pure disk based index storage
wouldn't be effected by this.)
--

2) currently we accomplish both of those transparently with a
transient-store configured to use a persistent-store.  we have some
subsystems which use a transient-store configured _not_ to use
persistence because of application requirements.

** is JCS (properly configured) a good way to meet this requirement?

CM: Yes.  You could very easily configure a separate role for transient
store which would use a different group / region (JCS terms.)  This
region would be configured to use _only_ memory store, with no support
for persistence.  This could be accomplished as is, by changing roles
appropriately, and providing different config in the xconf for the
transient store.
--

3) we have one subsystem which goes direct to the persistent-store right

at shutdown which could just as easily go to the in-memory front-end as
long as the persistence between restarts still happened.

** is JCS (properly configured) a good way to meet this requirement?

CM: Yes.  Again, this could either be written to the default store,
which would go to MRU, and then disk at shutdown, _or_ a different group
/ region could be created and configured which would go straight to
disk, without an in-memory MRU involved.
--

I am not following the details well here about the persistence issue. 
If we don't use the Disk Cache, what other persistent (on disk, survives

restarts) options are there?

CM: As I mentioned before, the only issue which needs more investigation
is abnormal shutdown with an in-memory index, however, this could
theoretically be up to the user to decide - if their application
requires an absolute guarantee of persistence at all times, they could
configure it to use an on-disk index at all times - this would require
no class change, just a change in the CCF.

Hope that answers your questions properly :)  I have a horrible feeling
that my email client won't format this nicely ;)
--

Geoff




>-----Original Message-----
>From: Unico Hommes [mailto:unico@hippo.nl]
>
>Sent: Saturday, 6 March 2004 3:46 a.m.
>To: dev@cocoon.apache.org
>Subject: Re: Turning off default MRU store
>
>
>Geoff Howard wrote:
>
> 
>
>>Unico Hommes wrote:
>>
>>   
>>
>>>Sylvain Wallez wrote:
>>>
>>>     
>>>
>>>>Unico Hommes wrote:
>>>>
>>>>       
>>>>
>>>>>Unico Hommes wrote:
>>>>>
>>>>>         
>>>>>
>>>>>>Corin Moss wrote:
>>>>>>
>>>>>>           
>>>>>>
>>>>>>>Hi Guys,
>>>>>>>
>>>>>>>I might be getting ahead of myself a bit here, but I'm going to
>>>>>>>try and turn off the default MRU store, in favour of the JCS
>>>>>>>             
>>>>>>>
>
> 
>
>>>>>>>based persistent store.  I'd like to try some tests on
>>>>>>>             
>>>>>>>
>
> 
>
>>>>>>>performance without the default MRU - has anyone else tried
>>>>>>>             
>>>>>>>
>
> 
>
>>>>>>>anything similar? I've simply set the core store's role to point
>>>>>>>             
>>>>>>>
>
> 
>
>>>>>>>to the JCS store implementation.
>>>>>>>
>>>>>>>             
>>>>>>>
>>>>>>I guess I already got ahead of you when I renamed
>>>>>>JCSPersistentStore JCSStore just now :-) (And merged it with the
>>>>>>           
>>>>>>
>
> 
>
>>>>>>AbstractJCSStore BTW). It seems to me that JCS is both and it
>>>>>>           
>>>>>>
>
> 
>
>>>>>>could replace all three stores: DefaultStore, TransientStore and
>>>>>>           
>>>>>>
>
> 
>
>>>>>>PersistentStore.
>>>>>>
>>>>>>           
>>>>>>
>>>>>I have to add that this is not the whole story. We do actually need

>>>>>to distinguish between persistent and transient storage. Some
>>>>>         
>>>>>
>
> 
>
>>>>>components explicitly want to persist some data instead of just
>>>>>         
>>>>>
>
> 
>
>>>>>cache it for speed. But as far as caching is concerned I think it
>>>>>         
>>>>>
>
> 
>
>>>>>we can leave it completely up to JCS.
>>>>>         
>>>>>
>>>>
>>>>
>>>>
>>>>Some components are explicitely using the transient-store to keep
>>>>data that shouldn't (or cannot) be serialized. But AFAIK, no
>>>>       
>>>>
>
> 
>
>>>>component directly uses the persistent-store except the store
>>>>       
>>>>
>itself.
> 
>
>>>To my knowlegde there is one in eventcache block that uses the
>>>PersistentStore to persist some info it needs to recover upon next
>>>     
>>>
>
> 
>
>>>startup.
>>>
>>>It does not look to me as though JCS would fit the PersistentStore
>>>role very well. I was thinking that perhaps. We could have JCSStore
>>>     
>>>
>
> 
>
>>>as a replacement for TransientStore and Store roles and use
>>>     
>>>
>
> 
>
>>>FilesystemStore for the PersistentStore role.
>>>     
>>>
>>
>>Wait a minute.  The original issue that brought JCS to the front as
>>that the persistent store was broken and had license problems.  Are
>>   
>>
>
> 
>
>>you saying that JCS isn't a good candidate for persisting the cached
>>   
>>
>
> 
>
>>info to disk, or that the fact that it by default has an MRU memory
>>   
>>
>
> 
>
>>front-end makes it not map directly to persistent-store role cleanly?
>>
>>   
>>
>IIUC, JCS will always use a memory store in front yes. And it suggests
>
>on the JCS website that the persistence process is not very reliable:
>
>from http://jakarta.apache.org/turbine/jcs/IndexedDiskAuxCache.html :
>
>"When the disk cache is properly shutdown, the memory index is written
>
>to disk and the value file is defragmented. When the cache starts up,
>
>the disk cache can be configured to read or delete the index file. This
>
>provides an unreliable persistence mechanism."
>
> 
>
>>The use of persistent store in eventcache shouldn't be a problem with
>>JCS as long as at shutdown, it persists everything it has in its MRU
>>   
>>
>
> 
>
>>if it hasn't already.  If there is some problem it would be much
>>   
>>
>
> 
>
>>better to just go back to the old serializing persistence for the
>>   
>>
>
> 
>
>>event cache data because the only benefit to putting it in the store
>>   
>>
>
> 
>
>>is that you more or less guaranteed that the events and cached
>>   
>>
>
> 
>
>>responses would be kept together.
>>   
>>
>
>
>Yes, giving up that feature would be too bad.
>
>Unico
> 
>


================================================================
CAUTION: This e-mail and any attachment(s) contains information
that is intended to be read only by the named recipient(s). It
may contain information that is confidential, proprietary or the
subject of legal privilege. This information is not to be used by
any other person and/or organisation. If you are not the intended
recipient, please advise us immediately and delete this e-mail
from your system. Do not use any information contained in it.

================================================================
For more information on the Television New Zealand Group, visit
us online at http://www.tvnz.co.nz
================================================================

Mime
View raw message