cocoon-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maarten Couvreur <mcouvr...@be-value.nl>
Subject Re: Caching a lot of large files problems
Date Tue, 05 Dec 2006 15:19:23 GMT
Ard Schrijvers <a.schrijvers <at> hippo.nl> writes:

> 
> Hello Maarten,
> 
> There is quite something to know about how to optimize your stores, but I will
try to help you out on every
> issue. If afterwards things are still not clear, let me know. The problem you
are having is pretty
> straighforward, and not to hard to fix, though needs quite some understanding
of cocoon's caching impl.
> 
> > 
> > 
> > Hi!
> > Please help me with the following problem, cause everything I 
> > try doesn't work.
> > 
> > We want to cache large xml-documents every night from an 
> > exist-database with
> > cocoon 2.1.9. Those files are about 600K each and we want to 
> > store them on
> > harddisk. The problem is that while genereating the cache, 
> > java runs out of
> > memory, depending on how much memory java has. It gives an error:
> > java.lang.OutOfMemoryError: Java heap space.
> > I would think that if memory is full, things should 
> > automatically be cleared
> > from cache in memory. But that doens't seem te happen. I've 
> > tried several
> > settings for java-memory but that all doesn't make a difference. 
> > 
> > This is the code in the sitemap for creating the cache:
> > 
> > <map:transform type="cinclude">
> >   <map:parameter name="source" value="d:/cache-dir"/> 
> >   <map:parameter name="support-caching" value="true"/>
> >   <map:parameter name="expires" value="172800"/>
> > </map:transform>
> 
> Never used the cinclude transformer like this (I always use the by default
caching IncludeTransformer,
> but since you are connecting to db, it is not cacheable by default.) One thing
I am not sure about, and I
> really need confirmation about it from you: Where do the cached items end up
in? In the transient store, or
> in the default store? How do you know this? Just start your app, and only call
the pipeline where this
> cinclude is in. Now, go to your status page (http://localhost:8888/status by
default, so just the
> hostname:port/status), and look where cache keys are created. If they are
added to transient store, we
> have to change that, if they are added to EHDefaultstore, it is ok
> 
> > 
> > 
> > The cocoon.xconf has the following settings:
> > <transient-store logger="core.store.transient">
> >   <parameter name="maxobjects" value="1000"/>
> >   <parameter name="use-persistent-cache" value="false"/>
> > </transient-store>
> > 
> > 
> > <store logger="core.store">
> > 	<parameter name="maxobjects" value="1000"/>
> > 	<parameter name="eternal" value="false"/>
> > 	<parameter name="timeToLiveSeconds" value="0"/>
> > 	<!-- 1 day -->
> > 	<parameter name="timeToIdleSeconds" value="86400"/>
> > 	<parameter name="overflow-to-disk" value="true"/>
> > 	<parameter name="diskpersistent" value="true"/>
> > 	<parameter name="use-persistent-cache" value="true"/>
> > </store>
> 
> As far as I know, use-persistent-cache is never used. diskpersistent means
persistent between jvm
> restarts. 
> 
> > 
> > And the following settings are in core.properties (the more 
> > memory I give, the
> > longer it takes before the memory error, but memory is at 
> > some point limited off
> > course):
> > store-janitor.freememory =  2000000
> > store-janitor.heapsize =   200000000
> > store-janitor.cleanup-thread-interval = 3
> > store-janitor.percent-to-free = 10
> 
> So, what is happening: you are having a heapsize of for example 200000000
(around 200 Mb, little less). How
> much memory does your JVM has? The heapsize should *always* be smaller then
the JVM maxmemory! Cocoon has a
> build in memory manager, the StoreJanitor, that tries to free memory when used
memory by the JVM exceeds
> the heapsize (the it tries to free percent_to_free). In my opinion, when this
StoreJanitor is needed to
> free memory, you will run into problems sooner or later anyway. So, the goal
of tuning you app is, that the
> StoreJanitor never tries to free memory.  
> 
> > 
> > I've also tried to change the jetty-settings, with no luck:
> > <Call name="addListener">
> >   <Arg>
> >     <New class="org.mortbay.http.SocketListener">
> >       <Set name="Port"><SystemProperty name="jetty.port" 
> > default="8888"/></Set>
> >       <Set name="MinThreads">10</Set>
> >       <Set name="MaxThreads">100</Set>
> >       <Set name="MaxIdleTimeMs">30000</Set>
> >       <Set name="LowResourcePersistTimeMs">5000</Set>
> >       <Set name="bufferSize">8192</Set>
> >     </New>
> >   </Arg>
> > </Call>
> 
> Do not look for a solution in the jetty-settings .
> 
> > 
> > EHcache is updated to ehcache-1.2.3.jar.
> > 
> > I hope someone can give some thoughts about were to look.
> 
> So, why do you get OOM? Suppose, you run with a JVM of 256 Mb max memory. Now,
you are caching your pipelines (I
> suppose all of them). 
> 
> So, suppose, you have a page, like this:
> 
> <map:pipeline type="caching">
> <map:match pattern="mypage">
> 	<map:aggregate>
> 		<map:part src="cocoon:/get-from-exist"/>
> 		<map:part src="cocoon:/someelse"/>
> 	</map:aggregate>
> 	<map:serialize type="xml"/>
> </map:match> 
> 
> Now, suppose, your cocoon:/get-from-exist is nicely cached, with the 600 kb
file. Then, the block above
> here, is cached as well when cocoon:/someelse is cacheable, so, for example
you already have ~1.2 Mb (2
> times 600). But, it might be possible that cocoon caches it many more times.
Anyway, even if cocoon caches
> it ones, you see that you have maxobjects in your store settings of 1000
items. So, suppose at some point,
> there are 500 cached items with the 600k files in it, that is 300 Mb!
maxobjects simple means : "how many
> cached responses should be kpet in memory". If your cached items end up in the
transient store, you cannot
> overflow them to disk (let me know, if in transientstore, then I can help you
out with another way, without
> the cinclude). If in ehdefaultstore, you can overflow items to disk. This is
what you want. Your
> maxobjects should be lowered in order to avoid your OOM. Bring them back to
150 or something, and try again
> (keep overflow-to-disk = true. when 150 is exceeded, cached responses are
overflowed to disk) . You can
> optimize settings just by testing: run xenu
(http://home.snafu.de/tilman/xenulink.html), a
> cralwer, and look at the status page what is happening with the memory.  
> 
> So, this is in short a little on caching. If you really want to have the best
solution, you have to do something
> similar I lately did, but it is not really trivial. From our remote
repository, I wanted binary data to be
> stored in a diskcache only, but normal files, like .xml documents, I want to
be kept in memory, for
> performance. Therefor, I have multiple stores in my cocoon app. Since we were
also using event cache, I had
> to build this, because it was not supported by default, but I think, when you
are not using event cache, you
> can configure multiple EHDefaultStores without problems. But, you need to know
how to configure all the
> roles, your map:pipes (multiple cache-roles) and your cocoon.xconf.
> 
> Hopefully, you won't be needing all this,
> 
> Regards Ard
> 
> > 
> > Regards, Maarten Couvreur
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe <at> cocoon.apache.org
> > For additional commands, e-mail: users-help <at> cocoon.apache.org
> > 
> > 
> 

Hi Ard,
thanx for your comments!

I made some changes in the settings.

I've set the following java-options:
set JAVA_OPTIONS= -Xmx800000000 -Xms800000000
(with 2 Gig of ram that should be ok)

I've changed the cocoon.properties as following:
transient-store.maxobjects = 1000
store.maxobjects = 150
store-janitor.freememory =  30000000
store-janitor.heapsize =   500000000

In the sitemap I've deleted the param:
<map:parameter name="source" value="d:/cache-dir"/> 

I've also changed the code a bit so all the data isn't stored in de pipeline.
The cached files indeed ended up in the EHdefaultstore. Your comment's where
very helpfull.

Now it works perfectly.

Thanx very much!




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org


Mime
View raw message