perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Wilga <>
Subject Re: BerkeleyDB as a cache backend
Date Mon, 27 Jun 2005 14:07:33 GMT
At 6:09 PM -0400 6/24/05, Arshavir Grigorian wrote:
>Hello list,
>I coded a caching system using BerkeleyDB::Hash as the backend. It 
>was working fine until the database file became fairly large (850M).
>At some point the performance degraded and the web server process 
>accessing the database started hanging. Someone suggested locking 
>issues being the cause for the hangups, but trying to access the db 
>from a single script even when there were no other processes 
>accessing it still hung.

Having used some pretty large (though not quite 850 Mb) BDB files, I 
can tell you my experiences. Unless you are using the fully 
transactional model, and have lots of disk space to throw at it, I'd 
now recommend against using BDB for anything that is updated from an 
httpd process.

The reason has to do with corruption. Even when using the Concurrent 
DB Store model, I found that I was spending a huge amount of time 
writing code to detect all the possible ways in which DB files can 
become corrupt when accessed directly by httpd.

After using BDB for several years, I recently re-coded everything to 
use replicated MySQL DBs. Not only is the Perl code much smaller now, 
it actually runs more quickly due to better indexing and the 
advantages that can be obtained by using SQL for calculations.

I have no doubt that BDB works very well for some things, but in my 
opinion an httpd process--with many concurrent threads that can be 
dropped unexpectedly at pretty much any time--is not one of them. 
(BDB is actually what the default MySQL DB format uses at its lowest 
level, but the corruption problem is avoided by having one, central 
DB daemon that does all of the reads/writes to the files.)

Dan Wilga                               
Web Administrator                   
Mount Holyoke College                                Tel: 413-538-3027
South Hadley, MA  01075            "Who left the cake out in the rain?"

View raw message