Mailing-List: contact modperl-help@perl.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (nike.apache.org: local policy)
User-Agent: Microsoft-Entourage/12.15.0.081119
Date: Tue, 20 Apr 2010 17:18:33 -0700
Subject: non-stop generational modperl config update strategies?
From: Jeff McCarrell <jmccarre@akamai.com>
To: <modperl@perl.apache.org>
Message-ID: <C7F39269.DD17%jmccarre@akamai.com>
Thread-Topic: non-stop generational modperl config update strategies?
Thread-Index: Acrg6C1EwQYN94cvBE2/qUi+utjw1Q==
Mime-version: 1.0
Content-type: multipart/alternative;
	boundary="B_3354628713_19438396"

> This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

--B_3354628713_19438396
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Hello modperl-folk:

In a nutshell: I would like to ask the community for pointers on how to
evolve our successfully
deployed application which gets restarted once / hour to reload
configuration state
to a model where it is continuously running.

Background:
We have a long-successfully-deployed mp2 application that has to scale well=
.
Our clusters are currently fronted by hardware load balancers.
To effect a configuration change, which is currently done hourly,
a process removes a cluster slave host from the hardware load balancers
rotation,
pushes the new config state down to the slave,
and restarts apache2.
After apache2 restarts, the mgmt process puts that host back into the load
balancer rotation.

We are running the prefork MPM with 32-512 httpd children on fairly beefy
linux boxes with 4 =AD 8 cores each
using the standard unix copy-on-write model:
load the config state into perl data structures in the parent once via a
PerlPostConfigHandler
then fork as many children as the prefork MPM decides it needs to handle th=
e
load.
The configuration pushes are scheduled via wall clock time;
currently it is deterministic when a configuration change will be fully
propagated.
All configuration changes are pushed to all slaves.
There are currently a few hundred slaves covering most of North America.
This model has worked and is working fairly well for us to date.

However, the clock-work nature of our scheme ultimately limits our
scalability.
And in thinking about what it would take to overcome our current
limitations,
I would like to be able to reload the configuration state from new data
without an apache restart.
Put another way, I would like to be able to load next generation
configuration state into new httpd children,
then kill off the previous generation as they complete their current
requests,
and start using the new instances, all while servicing requests,
albeit at perhaps a reduced rate while the configuration state is being
swapped.
=20
I don=B9t know how to do this in the prefork MPM;
and what I am proposing more-or-less breaks the load-parent / fork model.

The size of the configuration data are not too large: lets say 10Meg or so,
but large enough that I prefer models that share this state among all httpd
children.

At this point, I'd be willing to consider other MPMs if they help me get
there.
I am willing to trade memory and CPU to achieve a non-stop apache instance.

So modperl experts: any pointers on prior art here?
I'd love to hear about strategies that have been shown to work in real life=
.


TIA,

-- jeff


--B_3354628713_19438396
Content-type: text/html;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

<HTML>
<HEAD>
<TITLE>non-stop generational modperl config update strategies?</TITLE>
</HEAD>
<BODY>
<FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:11pt=
'>Hello modperl-folk:<BR>
<BR>
In a nutshell: I would like to ask the community for pointers on how to evo=
lve our successfully <BR>
deployed application which gets restarted once / hour to reload configurati=
on state<BR>
to a model where it is continuously running.<BR>
<BR>
Background:<BR>
We have a long-successfully-deployed mp2 application that has to scale well=
.<BR>
Our clusters are currently fronted by hardware load balancers.<BR>
To effect a configuration change, which is currently done hourly, <BR>
a process removes a cluster slave host from the hardware load balancers rot=
ation,<BR>
pushes the new config state down to the slave,<BR>
and restarts apache2.<BR>
After apache2 restarts, the mgmt process puts that host back into the load =
balancer rotation.<BR>
<BR>
We are running the prefork MPM with 32-512 httpd children on fairly beefy l=
inux boxes with 4 &#8211; 8 cores each<BR>
using the standard unix copy-on-write model:<BR>
load the config state into perl data structures in the parent once via a Pe=
rlPostConfigHandler <BR>
then fork as many children as the prefork MPM decides it needs to handle th=
e load.<BR>
The configuration pushes are scheduled via wall clock time; <BR>
currently it is deterministic when a configuration change will be fully pro=
pagated.<BR>
All configuration changes are pushed to all slaves.<BR>
There are currently a few hundred slaves covering most of North America.<BR=
>
This model has worked and is working fairly well for us to date.<BR>
<BR>
However, the clock-work nature of our scheme ultimately limits our scalabil=
ity.<BR>
And in thinking about what it would take to overcome our current limitation=
s,<BR>
I would like to be able to reload the configuration state from new data wit=
hout an apache restart.<BR>
Put another way, I would like to be able to load next generation configurat=
ion state into new httpd children,<BR>
then kill off the previous generation as they complete their current reques=
ts,<BR>
and start using the new instances, all while servicing requests, <BR>
albeit at perhaps a reduced rate while the configuration state is being swa=
pped.<BR>
&nbsp;<BR>
I don&#8217;t know how to do this in the prefork MPM;<BR>
and what I am proposing more-or-less breaks the load-parent / fork model.<B=
R>
<BR>
The size of the configuration data are not too large: lets say 10Meg or so,=
<BR>
but large enough that I prefer models that share this state among all httpd=
 children.<BR>
<BR>
At this point, I'd be willing to consider other MPMs if they help me get th=
ere.<BR>
I am willing to trade memory and CPU to achieve a non-stop apache instance.=
<BR>
<BR>
So modperl experts: any pointers on prior art here?<BR>
I'd love to hear about strategies that have been shown to work in real life=
.<BR>
<BR>
<BR>
TIA,<BR>
<BR>
-- jeff<BR>
</SPAN></FONT>
</BODY>
</HTML>


--B_3354628713_19438396--