cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reinhard Poetz <>
Subject Re: StoreJanitor
Date Tue, 03 Apr 2007 20:34:59 GMT
Ard Schrijvers wrote:
 > Before I say things that are wrong, please consider that the StoreJanitor was 
invented long before I looked into the cocoon code, so probably a lot of 
discussion and good ideas has been around which I am not aware of. But still, my 
ideas about the StoreJanitor (and sorry for the long mail, but perhaps it might 
contain something useful):
 > 1) How it works and its intention (I think :-) ):  The StoreJanitor is 
originally invented to monitor cocoon's memory useage and does this by checking 
some memory values every X (default 10) seconds. Beside the fact that I doubt 
users know that it is quite important to configure the store janitor correctly, 
I stick to the defaults and use a heapsize of just a little lower then JVM 
 > Now, every 10 seconds, the StoreJanitor does a check wether 
(getJVM().totalMemory() >= getMaxHeapSize() && (getJVM().freeMemory() < 
getMinFreeMemory()) is true, and if so, the next store is choosen (compared to 
previoud one) and entries are removed from this store (I saw a post that in 
trunk not one single store is chosen anymore, but an equal part of all of them 
is being removed, right?...probably you can configure which stores to use, i 
don't know)

AFAICS there are two freeing algorithms in trunk: round-robin and all-stores.

 > 2) My Observations: When running high traffic sites and render them live 
(only mod_cache in between which holds pages for 5 to 10 min) like [1] or [2], 
then checking every X sec for a JVM to be low on memory doesn't make sense to 
me. At the moment of checking, the JVM might be perfectly sound but just needed 
some extra memory for a moment, in that case, the Store Janitor is removing 
items from cache while not needed. Also, when the JVM is really in trouble, but 
the Store Janitor is not checking for 5 more sec....this might be too long for a 
JVM in a high traffic site when it is low on memory. Problems that result from 
it are:
 > - Since there is no way to remove cache entries from the used cache impl by 
the cache's eviction policy, the cache entries from memory are removed by 
starting from entry 0, whatever this might be in the cache. There is a very 
likely situation, that at the very next request, the same cache entries are 
added again.
 > - Ones the JVM gets low on memory, and the StoreJanitor is needed, it is 
quite likely that from that moment on, the StoreJanitor runs *every* 10 seconds, 
and keeps removing cache entries which you perhaps don't want to be removed, 
like compiled stylesheets.

yep, that's a problem

 > 	1) suppose, from one store (or since trunk from multiple stores) 10% 
(default) is removed. This 10% is from the number of memory cache entries. I 
quite frequently happen to have only 200 entries in memory for each store ( I 
have added *many* different stores to enable all we wanted in a high traffic 
environment) and the rest is disk store. Now, suppose, the JVM which has 512 Mb 
of memory, is low on memory, and removes 10% of 200 entries = 20 entries, 
helping me zero!


 > These memory entries are my most important ones, so, on the next request, 
they are either added again, or, from diskcache I have a hit, implying that the 
cache will put this cache entry in memory again. If I would use 2000 memory 
items, I am very sure, the 200 items which are cleaned are put back in memory 
before the next StoreJanitor runs.
 > 	2) I am not sure if in trunk you can configure wether the StoreJanitor 
should leave one store alone, like the DefaultTransientStore. In this store, 
typically, compiled stylesheets end up, and i18n resource bundles. Since these 
files are needed virtually on every request, I had rather not that the 
StoreJanitor removes from this store. I think, the StoreJanitor does so, leaving 
my "critical app" in an even worse state, and on the next request, the hardly 
improved JVM needs to recompile stylesheets and i18n resource bundles.


 > 	3) What if the JVM being low is not because of the stores....For example, 
you have added some component which has some problems you did not know, and, 
that component is the real reason for you OOM. The StoreJanitor, sees your low 
memory, and starts removing entries from your perfectly sound cache, leaving you 
app in a much worse situation then it already was. Your component with memory 
leak has some more memory it now can fill, and hapily does this, making the 
StoreJanitor remove more and more entries from cache, untill it ends up with an 
empty cache. You could blame the wrong component for this behavior. One of these 
wrong components in use is the event registry for event caching, which made our 
high traffic sites with 512 Mb crash every two days. Better that I write in 
another mail what I did to the event cache registry, why I did not yet post 
about it, and if others are interested and how to include it in the trunk. 
Bottom line is that there was a major OOM problem if the registry grows, 
resulting in a StoreJanitor removing cache entries while this was actually 
increasing the problem.
 > 	4) By default, probably most people are using ehcache. Naturally, 
overflow-to-disk is true. In a high traffic site, the number of cache keys can 
grow enormously (I have seen mails around people complaing about disk cached 
growing to multiple Gbytes). Certainly, when the not very experienced user uses 
something like a session attr (or timestamp and many more possibilities) in a 
stylesheet parameter which ends up in the cache key (but perhaps, should cocoon 
be the target for high traffic sites for the average user, I don't know). Now, 
and this is IMO one of the major weakenesses of ehcache (or I missed it 
completely), I did not find any way to limit the number of disk store entries.

Actually we don't configure this value. According to the default 
value is 0 meaning unlimited. We should use the 1.2.4 constructor that allows to 
set a maxElementsOnDisk parameter.

 > This implies, that the disk store can grow indefinitely. For the ones ever 
looking at the status page, cache keys in memory of about 2 kb are quite common 
in cocoon (actually, the dept of the folder structure of your app is of 
influence). The disk store cache keys are kept in *memory*. So, suppose, you run 
your app with 128 Mb, and you have overflow-to-disk=true, your app runs into 
problem when there are about 50.000 keys in cache. Then your StoreJanitor keep 
removing entries from your memory cache, which are refilled with disk store 
entries just a few moments later. Now, if you really know how to configure your 
stores, you use a time2liveSeconds and time2IdleSeconds to let your store clear 
unused cache entries. This is good to do, unless, you depend on something like 
an event registry which is currently in cocoon trunk. The problem is, that the 
StoreJanitor removes cache entries by calling the free from the correct store, 
which, might for example be the eventaware store. This event aware store, 
updates (cleans) its registry before removing the cache entry from its delegate. 
Now, when you use the internal cleaning of caches by a time2liveSeconds or 
time2IdleSeconds, the event registry is not cleaned and will lead to OOM in the 
long run.
 > I have more things about it, but probably nobody will read it anymore, but in 
short, my conclusion is that the StoreJanitor never helped me out, but merely 
impoverished my app when it ran

I wonder what StoreJanitor is good for at all. EHCache takes care that the 
number of items in the memory cache doesn't grow indefinitly and starts its own 
cleanup threads for the disc store 
JCS will probably do the same. I guess that original purpose of StoreJanitor was 
when Cocoon had its own store implementations (transient, persistent) and we had 
to take care of cleaning them up in our code.
Only the persistent store can grow unlimited but since it should only be used 
for special usecases, it shouldn't be a real problem.

 > 						--------o0o--------
 > The rules I try to follow to avoid the Store Janitor to run
 > 1) use readers in noncaching pipelines and use expires on them to avoid 
cache/memory polution
 > 2) use a different store for repository binary sources which has only a disk 
store part and no memory part (cached-binary: protocol added)
 > 3) use a different store for repository sources then for pipeline cache
 > 4) replaced the abstract double mapping event registry to use weakreferences 
and let the JVM clean up my event registry
 > 5)  (4) gave me undesired behavior by removing weakrefs in combination with 
ehcache when overflowing items to disk (i could not reproduce this, but seems 
that my references to cachekeys got lost). Testing with JCSCache solved this 
problem, gave me faster response times and gave me for free to limit the number 
of disk cache entries. Disadvantage of the weakreferences, is that I disabled 
persitstent caches for jvm restarts, but, I never wanted this anyway (but this 
might be implemented quite easily, but might take long start up times)
 > 6) JCSCache has a complex configuration IMO. Therefor, I added default 
configurations to choose from, for example:
 > <store logger="">
 >    <parameter name="region-name" value="store"/>
 >    <parameter name="size" value="small"/>
 > where size might be small, medium, large or huge.
 > I think we have created with in this way a setup for cocoon, where it is 
harder for unexperienced users to have memory > problems when trying to 
implement larger sites.
 > Hopefully somebody read my mail until here  :-)  I am curious about what 
others think,
 > [1]
 > [2]

What do we want to do in order to improve the situation? After reading your mail 
and from my own experience I'd say

  - introduce a maxPersistentObjects parameter and use it in EHDefaultCache to
    set maxElementsOnDisk
  - make the registration of stores at StoreJanitor configureable
    (Though I wonder what the default value should be, true or false?)
  - fix EventRegistry

Any further ideas?

P.S. Ard, answering to your mails is very difficult because there are no line 
breaks. Is anybody else experiencing the same problem or is it only me?

Reinhard Pötz           Independent Consultant, Trainer & (IT)-Coach 

{Software Engineering, Open Source, Web Applications, Apache Cocoon}


View raw message