httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <>
Subject Re: Fwd: svn commit: r1228700 - in /httpd/httpd/trunk/docs: STATUS manual/mod/mod_heartbeat.xml manual/mod/mod_heartmonitor.xml manual/mod/mod_watchdog.xml manual/mod/mod_watchdog.xml.meta
Date Sat, 07 Jan 2012 21:08:33 GMT
On 07.01.2012 20:22, Sander Temme wrote:
> Folks,
> I've put in some updates to the mod_heart* modules, taken from the README supplied with
the modules and some perusal of the source code.  I have not run these as I'm too lazy to
set up the servers.  Review would be appreciated, especially by the original authors (Cc),
before backport to 2.4.x.

I added some comments inline, just based on reading your docs.

> Also added a doc file for mod_watchdog.  Question for the docs folks: since this one
is only of interest for consumption by other modules, how much documentation should we supply
in the user-facing pages?  I said "this module defines programmatic hooks"... do we need to
define what the hook handler prototypes are?  That's more like developer documentation.  Thoughts?

I don't know about the history of similar decisions, but IMHO what you 
wrote is good enough for 2.4.0.

>> Modified: httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml
>> URL:
>> ==============================================================================
>> --- httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml (original)
>> +++ httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml Sat Jan  7 19:10:28 2012
>> @@ -35,21 +35,66 @@ that advertises the servers current conn
>> will be running on a proxy server with<module>mod_lbmethod_heartbeat</module>
 loaded, which allows
>> <directive module="mod_proxy">ProxyPass</directive>  to use the "heartbeat"<em>lbmethod</em>
>> of<directive module="mod_proxy">ProxyPass</directive>.</p>
>> +<p>
>> +<module>mod_heartbeat</module>  itself is loaded on the origin server(s)
that serve requests
>> +    through the proxy server(s).
>> +</p>
>> +
>> +<note type="warning">
>> +        To use<module>mod_heartbeat</module>,
>> +<module>mod_status</module>  and<module>mod_watchdog</module>
>> +        must be either a static modules or, if a dynamic module, it must
>> +        be loaded before<module>mod_heartbeat</module>.
>> +</note>
>> +
>> </summary>
>> +<section id="consuming">
>> +<title>Consuming mod_heartbeat Output</title>
>> +<p>
>> +      Every 1 second, this module generates a single multicast UDP
>> +      packet, containing the number of busy and idle workers.  The
>> +      packet is a simple ASCII format, similiar to GET query parameters
>> +      in HTTP.
>> +</p>
>> +
>> +<example><title>An Example Packet</title>
>> +v=1&amp;ready=75&amp;busy=0
>> +</example>
>> +
>> +<p>
>> +    Consumers should handle new variables besides busy and ready,
>> +    separated by '&amp;', being added in the future.
>> +</p>
>> +
>> +</section>
>> +
>> +<section id="miscellaneous">
>> +<title>Miscellaneous</title>
>> +<p>The interval of 1 seconds is controlled by the HEARTBEAT_INTERVAL
>> +compile time define.  This is not currently tunable at run time. To make
>> +this module send the status packet more often, you must add to the
>> +CFLAGS used to compile the module to include:</p>
>> +<example>-DHEARTBEAT_INTERVAL=3</example>
>> +<p>Would cause the broadcasts to be sent every 3 seconds.</p>
>> +</section>

"more often" than 1 second does not match the example "3 seconds" ;)
I guess more often is not possible (if resolution is seconds), so it 
would be "less often"?

More important: I think this is totally wrong. The define exists, but I 
couldn't find any place in the code, which is influenced by it. I 
*guess* it uses the global watchdog interval configured for mod_watchdog.

>> <directivesynopsis>
>> <name>HeartbeatAddress</name>
>> -<description>Address to send heartbeat requests</description>
>> +<description>Multicast address for heartbeat packets</description>
>> <syntax>HeartbeatAddress<var>addr:port</var></syntax>
>> <default>disabled</default>
>> <contextlist><context>server config</context></contextlist>
>> <usage>
>> -<p>The<directive>HeartbeatAddress</directive>  directive specifies
>> -    address<module>mod_heartbeat</module>  will send status information
to. This
>> -    address will usually corrspond to a configured<directive
>> -    module="mod_heartmonitor">HeartbeatListen</directive>  on a frontend
>> -    proxy system.</p>
>> +<p>The<directive>HeartbeatAddress</directive>  directive specifies
>> +multicast address to which<module>mod_heartbeat</module>  will send
>> +status information. This address will usually correspond to a configured
>> +<directive module="mod_heartmonitor">HeartbeatListen</directive>  on
>> +frontend proxy system.</p>
>> +<example>
>> +HeartbeatAddress
>> +</example>
>> </usage>
>> </directivesynopsis>

Should we note, that Multicast needs to be enabled on both notes and 
routing needs to be set up, so that the multicast packets from the 
origin server can reach the proxy? On the one hand it is obvious, on the 
other hand not unlikely to be the typical first problem people run into.

>> Modified: httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml
>> URL:
>> ==============================================================================
>> --- httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml (original)
>> +++ httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml Sat Jan  7 19:10:28 2012
>> @@ -20,19 +20,19 @@
>>   limitations under the License.
>> -->
>> -<modulesynopsis metafile="mod_heartbeat.xml.meta">
>> +<modulesynopsis metafile="mod_heartmonitor.xml.meta">
>> <name>mod_heartmonitor</name>
>> <description>centralized monitor for mod_heartbeat origin servers</description>
>> <status>Experimental</status>
>> -<sourcefile>mod_heartmonitor</sourcefile>
>> +<sourcefile>mod_heartmonitor.c</sourcefile>
>> <identifier>heartmonitor_module</identifier>
>> <compatibility>Available in Apache 2.3 and later</compatibility>
>> <summary>
>> <p>
>> <module>mod_heartmonitor</module>  listens for server status messages
>> -by<module>mod_heartbeat</module>  enabled servers and makes their status
>> +by<module>mod_heartbeat</module>  enabled origin servers and makes their
>> available to<module>mod_lbmethod_heartbeat</module>. This allows
>> <directive module="mod_proxy">ProxyPass</directive>  to use the "heartbeat"
>> <em>lbmethod</em>  inside of<directive module="mod_proxy">ProxyPass</directive>.
>> @@ -41,22 +41,34 @@ available to<module>mod_lbmethod_heartb
>> <p>This module uses the services of<module>mod_slotmem_shm</module>
>> available instead of flat-file storage.  No configuration is required to
>> use<module>mod_slotmem_shm</module>.</p>
>> +
>> +<note type="warning">
>> +        To use<module>mod_heartmonitor</module>,
>> +<module>mod_status</module>  and<module>mod_watchdog</module>
>> +        must be either a static modules or, if a dynamic module, it must
>> +        be loaded before<module>mod_heartmonitor</module>.
>> +</note>
>> +


>> </summary>
>> <directivesynopsis>
>> <name>HeartbeatListen</name>
>> -<description>address to listen for incoming heartbeat requests</description>
>> +<description>multicast address to listen for incoming heartbeat requests</description>
>> <syntax>HeartbeatListen<var>addr:port</var></syntax>
>> <default>disabled</default>
>> <contextlist><context>server config</context></contextlist>
>> <usage>
>>      <p>The<directive>HeartbeatListen</directive>  directive specifies
>> -    address the server will listen on for status information from
>> +    multicast address on which the server will listen for status information from
>>      <module>mod_heartbeat</module>-enabled servers.  This
>>      address will usually corrspond to a configured<directive
>>      module="mod_heartbeat">HeartbeatAddress</directive>  on an origin server.
>>      </p>
>> +
>> +<example>
>> +    HeartbeatListen
>> +</example>

Same remark concerning the use of multicast as above.

>>      <p>  This module is inactive until this directive is used.</p>
>> </usage>
>> @@ -88,7 +100,7 @@ heartbeat requests to this server</descr
>>      <p>The<directive>HeartbeatMaxServers</directive>  directive
specifies the
>>      maximum number of servers that will be sending requests to this monitor
>>      server.  It is used to control the size of the shared memory allocated
>> -    to store the heartbeat info.</p>
>> +    to store the heartbeat info when<module>mod_slotmem_shm</module>
 is in use.</p>

Can this be changed using graceful restart? Only for flat file or also 
for shm?

Typo: "corrspond" -> "correspond"

Additional info: it seems the data used for the load balancer is updated 
about every 5 seconds (define HM_UPDATE_SEC). More precisely it seems 
the heartmonitor is called by watchdog (with interval 0 seconds, but 
watchdog sleep at least 100ms), then updates the data from what is had 
previously read, then loops for 5 seconds reading new data and then 
returns to watchdog. This is done one one thread per child, but a mutex 
guarantees, that it only runs in one child at every time. Hope I'm right.

Furthermore the heartmonitor contains some additional code to handle 
POST requests directed to it and reading usage data directly from the 
POST body (in addition to the multicast method).

I think the current info in mod_lbmethod_heartbeat.xml is confusing. It 

"This modules load balancing algorithm favors servers with more ready 
(idle) capacity over time, but does not select the server with the most 
ready capacity every time.  Servers that have 0 active clients are 
penalized, with the assumption that they are not fully initialized."

As far as I can see in the code, it randomly chooses one of the idle 
origin server slots based on the last received heartbeat data. Another 
equivalent formulation would be: it does a weighted balancing to the 
origin servers, where the weights are proportional to the idle (ready) 
slots reported with the last heartbeat message.

I couldn't find code that penalizes servers with 0 active clients.

It seems the heartbeat lb opens the data file for each request. So if 
I'm right, using slotmem is highly recommended for performance reasons.

>> </usage>
>> </directivesynopsis>
>> </modulesynopsis>
>> Added: httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml
>> URL:
>> ==============================================================================
>> --- httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml (added)
>> +++ httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml Sat Jan  7 19:10:28 2012
>> @@ -0,0 +1,60 @@
>> +<?xml version="1.0"?>
>> +<!DOCTYPE modulesynopsis SYSTEM "../style/modulesynopsis.dtd">
>> +<?xml-stylesheet type="text/xsl" href="../style/manual.en.xsl"?>
>> +
>> +<!--
>> + Licensed to the Apache Software Foundation (ASF) under one or more
>> + contributor license agreements.  See the NOTICE file distributed with
>> + this work for additional information regarding copyright ownership.
>> + The ASF licenses this file to You under the Apache License, Version 2.0
>> + (the "License"); you may not use this file except in compliance with
>> + the License.  You may obtain a copy of the License at
>> +
>> +
>> +
>> + Unless required by applicable law or agreed to in writing, software
>> + distributed under the License is distributed on an "AS IS" BASIS,
>> + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> + See the License for the specific language governing permissions and
>> + limitations under the License.
>> +-->
>> +
>> +<modulesynopsis metafile="mod_watchdog.xml.meta">
>> +<name>mod_watchdog</name>
>> +<description>provides infrastructure for other modules to periodically run
>> +    tasks</description>
>> +<status>Base</status>
>> +<sourcefile>mod_watchdog.c</sourcefile>
>> +<identifier>watchdog_module</identifier>
>> +<compatibility>Available in Apache 2.3 and later</compatibility>
>> +
>> +<summary>
>> +<p><module>mod_watchdog</module>  defines programmatic hooks for
other modules to
>> +periodically run tasks.  These modules can register handlers for
>> +<module>mod_watchdog</module>  hooks.  Currently, the following modules
in the
>> +Apache distribution use this functionality:</p>
>> +<ul>
>> +<li><module>mod_heartbeat</module></li>
>> +<li><module>mod_heartmonitor</module></li>
>> +</ul>
>> +<note type="warning">
>> +To allow a module to use<module>mod_watchdog</module>  functionality,
>> +<module>mod_watchdog</module>  itself must be statically linked to the
>> +core or, if a dynamic module, be loaded before the calling module.
>> +</note>
>> +</summary>
>> +
>> +<directivesynopsis>
>> +<name>WatchdogInterval</name>
>> +<description>Watchdog interval in seconds</description>
>> +<syntax>WatchdogInterval<var>number-of-seconds</var></syntax>
>> +<default>WatchdogInterval 1</default>
>> +<contextlist><context>server config</context></contextlist>
>> +
>> +<usage>
>> +Sets the interval at which the watchdog_step hook runs.  Default is to run every
>> +second.
>> +</usage>
>> +</directivesynopsis>
>> +</modulesynopsis>
>> +

We could mention, that there are two ways (and APIs) a module can use 
mod_watchdog. Either synchronously called every watchdog interval 
seconds, or running on a separate thread called every interval seconds, 
where interval is given during the setup.

Furthermore modules can tell mod_watchdog, whether the tasks should be 
run in each child independently, or coordinated by a mutex.



View raw message