Mailing-List: contact modperl-help@perl.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (asf.osuosl.org: local policy)
In-Reply-To: <4b3372230603072116y67b07720j90a8041d07eefa55@mail.gmail.com>
References: <4b3372230603072105l30e34a28v6861ddd209597f0c@mail.gmail.com>
 <4b3372230603072116y67b07720j90a8041d07eefa55@mail.gmail.com>
Mime-Version: 1.0 (Apple Message framework v746.2)
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <D34C5ADA-5C54-44F2-B957-CB434BF1CFA5@2xlp.com>
Cc: modperl@perl.apache.org
Content-Transfer-Encoding: 7bit
From: Jonathan Vanasco <modperl-list@2xlp.com>
Subject: Re: changing global data strategy
Date: Wed, 8 Mar 2006 01:34:20 -0500
To: Will Fould <willfould@gmail.com>


how big are these data structures?

200k?  2mb?  20mb?

if they're not too big, you could just use memcached.

	http://danga.com:80/memcached/
	http://search.cpan.org/~bradfitz/Cache-Memcached-1.15/Memcached.pm

its ridiculously painless to implement. i found it easier that a lot  
of other approaches.	

but if you have 50mb of data, i'd rethink what you're doing.

you're just going to keep getting screwed when your cache db updates  
(because the updates will only be per-child, not per parent  
process).  so you've got the potential for a 50MB  parent process  
having children that read in 50mb each in data?  thats a cascading  
nightmare.

if you need to precache such giant data structures, i'd do something  
like 2 tiered server
	apache a - talks to web users / load balancer
				sends data / whatever for specific processing to
	daemon b - either apache or some custom server, which handles  
precaching of db and parsing requests from apache
	db - datastore

having all of that data in modperl would be a nightmare though.  even  
with memcached, you'll update fast and everyone can access it, but  
you're going to keep eating memory.  if every session is going to  
toss though 20mb hashes of info, i'd keep that info out of apache  
entirely


On Mar 8, 2006, at 12:16 AM, Will Fould wrote:

> at this point, the application is on a single machine, but I'm  
> being tasked with moving our database onto another machine and  
> implement load balancing b/w 2 webservers.
>
> william
>
>
> On 3/7/06, Will Fould <willfould@gmail.com> wrote:
> an old issue:
>    "a dream solution would be if all child processes could *update*  
> a large global structure."
>
> we have a tool that loads a huge store of data (25-50Mb+) from a  
> database into many perl hashes at start up: each session needs  
> access to all these data but it would be prohibitive to use mysql  
> or another databases for multiple, large lookups (and builds), at  
> each session:  there are quite a few structures, each are very big.
>
> if the data never changed, it would be trivial; load/build just at  
> start-up.
>
> but since the data changes often, we use a semaphore strategy to  
> determine when childern should reload/rebuild the structures (after  
> updates have been made).
>
> this is painful. there has got to be a better way of doing this -  
> I've seen posts on memcache and other, more exotic animals.
>
> can someone point me in the right direction: a reference/read, or a  
> stable modules that exist for our situation?