perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff McCarrell <jmcca...@akamai.com>
Subject non-stop generational modperl config update strategies?
Date Wed, 21 Apr 2010 00:18:33 GMT
Hello modperl-folk:

In a nutshell: I would like to ask the community for pointers on how to
evolve our successfully
deployed application which gets restarted once / hour to reload
configuration state
to a model where it is continuously running.

Background:
We have a long-successfully-deployed mp2 application that has to scale well.
Our clusters are currently fronted by hardware load balancers.
To effect a configuration change, which is currently done hourly,
a process removes a cluster slave host from the hardware load balancers
rotation,
pushes the new config state down to the slave,
and restarts apache2.
After apache2 restarts, the mgmt process puts that host back into the load
balancer rotation.

We are running the prefork MPM with 32-512 httpd children on fairly beefy
linux boxes with 4 ­ 8 cores each
using the standard unix copy-on-write model:
load the config state into perl data structures in the parent once via a
PerlPostConfigHandler
then fork as many children as the prefork MPM decides it needs to handle the
load.
The configuration pushes are scheduled via wall clock time;
currently it is deterministic when a configuration change will be fully
propagated.
All configuration changes are pushed to all slaves.
There are currently a few hundred slaves covering most of North America.
This model has worked and is working fairly well for us to date.

However, the clock-work nature of our scheme ultimately limits our
scalability.
And in thinking about what it would take to overcome our current
limitations,
I would like to be able to reload the configuration state from new data
without an apache restart.
Put another way, I would like to be able to load next generation
configuration state into new httpd children,
then kill off the previous generation as they complete their current
requests,
and start using the new instances, all while servicing requests,
albeit at perhaps a reduced rate while the configuration state is being
swapped.
 
I don¹t know how to do this in the prefork MPM;
and what I am proposing more-or-less breaks the load-parent / fork model.

The size of the configuration data are not too large: lets say 10Meg or so,
but large enough that I prefer models that share this state among all httpd
children.

At this point, I'd be willing to consider other MPMs if they help me get
there.
I am willing to trade memory and CPU to achieve a non-stop apache instance.

So modperl experts: any pointers on prior art here?
I'd love to hear about strategies that have been shown to work in real life.


TIA,

-- jeff


Mime
View raw message