tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Anderson <tander...@infonow.com>
Subject Re: JDBCStore Downgrade (Tomcat4 vs. Tomcat 5)
Date Thu, 06 Jan 2005 00:28:19 GMT
It looks like the list stripped the image attachments.   Let me know if 
you're interested and I'll forward you the original email with the 
pictures.

~Tom

On Jan 5, 2005, at 4:59 PM, Tom Anderson wrote:

> FYI,
>
> We applied the 2 patches described in this thread with dramatic 
> effects on our network bandwidth.   Our JDBCStore talks to a database 
> on another machine and this graph shows how the bandwidth usage 
> dropped dramatically after these patches went up last night at 17:00 
> (most recent time is on the LEFT):
>
>
>
> I beleive that green represents bytes/sec FROM the database machine 
> and the blue represents bytes/sec TO the database.  The graph over the 
> last week shows the same thing:
>
>
>
> I hope this picture really illustrates the problem and demonstrates 
> the worthiness of the patch.   Again, I volunteer to make the patch to 
> whatever branches it's needed on.
>
> ~Tom
>
> p.s. I hope sending a picture to the list isn't taboo... if so, sorry.
>
> On Dec 30, 2004, at 8:53 PM, Tom Anderson wrote:
>
>> And to fix the loading of too many sessions while processing expires, 
>> I suggest the attached patch to only load in sessions that you KNOW 
>> need to be expired (I believe it's better to err in loading too few 
>> sessions during store expiration since they will get removed 
>> eventually anyways).   Here's that patch:
>>
>> <patch2.txt>
>>
>> FYI, I am testing both patches in my development environment and they 
>> seem to work well and achieve what I need to go forward with Tomcat 
>> 5.   Hopefully you will accept one or both to make Tomcat more 
>> robust!   Let me know if you want me to apply either or both patches 
>> to the HEAD (or whatever).   And I would also be willing to do doc 
>> patches as needed.
>>
>> Thanks,
>> ~Tom
>>
>> On Dec 29, 2004, at 7:16 PM, Tom Anderson wrote:
>>
>>> I would like to share my experiences in upgrading from Tomcat 4 to 
>>> Tomcat 5 and offer some suggestions for improvement (to Tomcat 5) 
>>> based on them.  This message got a little long but I hope somebody 
>>> in the know will find the time to read it and help me figure out how 
>>> to rectify my problems so I can upgrade to Tomcat 5 (again).
>>>
>>> BACKGROUND
>>>
>>> We have been using Tomcat 4 successfully for a couple of years (or 
>>> more?).   On some machines we have quite a few contexts running 
>>> (approx. 75) so making sure they all work well together is 
>>> important.    We have been using the "experimental" JDBCStore 
>>> mechanism with very good results and I have also contributed some 
>>> patches to achieve this success.
>>>
>>> One problem we discovered early on with Tomcat 4 was the 
>>> JDBCStore.processExpires() being called on all of our many contexts 
>>> at roughly the same time.   The effect was that the database would 
>>> slow and/or crash.  Fortunately, we were able to work around this 
>>> problem using the checkInterval settings in JDBCStore.   By 
>>> staggering these checkIntervals to different values we could ensure 
>>> that different contexts would hit the database at various times and 
>>> spread the load out.   Also, in our installation, the JDBCStore 
>>> checkInterval has been set about 15 times higher than the 
>>> PersistentManager checkInterval because it is slower and simply 
>>> doesn't need to be run as often (clearing out the store isn't as 
>>> important or fast as clearing out memory).
>>>
>>> TOMCAT 5
>>>
>>> Recently we tried to upgrade to Tomcat 5 and discovered that our 
>>> database was periodically choking and causing massive problems with 
>>> Tomcat's ability to process requests.   So we have had to rollback 
>>> to Tomcat 4.   I have spent the last several days going through the 
>>> Tomcat 5 code to try to figure out what has been happening.
>>>
>>> I have discovered that the checkInterval values are no longer used 
>>> (despite what the documentation says) and are instead replaced by 
>>> the backgroundProcessorDelay value in the Container class.   This 
>>> means that I now only have one value to set for the equivalent of 
>>> the 2 checkInterval values we had in Tomcat 4.   I suspect that this 
>>> was done to decrease the number of threads necessary which seems 
>>> like a noble goal.   But the downside is that I now am forced to run 
>>> my JDBCStore.processExpires() much more often than I'd like (15 
>>> times more often in my case) which is a hardship on the database.
>>>
>>> Furthermore, in Tomcat 4 JDBCStore overrode 
>>> StoreBase.processExpires() to only select older sessions from the 
>>> database something like this:
>>> 	SELECT id FROM tomcat_sessions
>>> 	WHERE app = '/Standalone/localhost/acme' AND 1104288671296 > 
>>> (lastaccess + maxinactive*1000)))
>>> This was a good thing because each session is loaded into memory 
>>> before deciding whether or not to expire it, so you don't want to 
>>> load every Stored session if possible.   In my case we often have 
>>> tens of thousands of sessions in Store so loading all those older 
>>> sessions every 60 seconds is quite a hardship, especially when 
>>> combined with the fact that it is running 15 times as often!
>>>
>>> To summarize the problems:
>>> 	1. no control of the StoreBase.processExpires() interval means that 
>>> processing expired persisted sessions is done as frequently as 
>>> sessions in memory which is too often in a production environment
>>> 	2. the processing of Stored expires is loading all old sessions 
>>> into memory before deciding which ones need to be removed from Store 
>>> instead of loading older sessions
>>> 	3. the documentation still suggests that the checkInterval values 
>>> are used in Tomcat 5
>>>
>>> SOLUTIONS:
>>>
>>> There are many ways to solve problem #1.   Here are my thoughts:
>>> 	1a. We could reinstate the checkInterval for the Store.   But I can 
>>> appreciate the desire to keep the number of threads down so maybe 
>>> this isn't the best idea.
>>> 	1b. Invent a new parameter in StoreBase, something like checkRatio 
>>> (TBR), that indicates how often to processExpires in relation to the 
>>> PersistentManager.   For example, if I set checkRatio="15" that 
>>> means only run StoreBase.processExpires on every 15th call to 
>>> PersistentManager.backgroundProcess().
>>> 	1c. Have ManagerBase and StoreBase both extend ContainerBase so 
>>> that we can set different backgroundProcessorDelay values for them.  
>>>  This one makes me a bit nervous because I'm not familiar with the 
>>> architectural issues it might raise but it seems like it could work.
>>>
>>> As for #2:
>>> 	2a. Override processExpires() in JDBCStore so that it only loads 
>>> sessions that have a good chance of needing expiration.
>>> 	2b. Define a new method in Store called something like oldKeys() 
>>> that returns keys that are past expiration and call that method in 
>>> StoreBase.processExpires()
>>>
>>> And the solution to #3 is obvious... remove "checkInterval" from the 
>>> documentation that appears here:
>>> 	http://jakarta.apache.org/tomcat/tomcat-5.0-doc/config/manager.html
>>>
>>> If you read this far, thank you very much.   I look forward to 
>>> hearing back from somebody who can help.   And I would be happy to 
>>> make patch(es) for the agreed-upon solution(s).
>>>
>>> ~Tom
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org


Mime
View raw message