Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
Subject: Re: Work in progress: mod_proxy Health Check module
From: Jim Jagielski <jim@jaguNET.com>
In-Reply-To: <56907444.2070905@primary.net>
Date: Sat, 9 Jan 2016 13:18:14 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <D9F39EF2-B1AC-4453-A854-33CBEFF42621@jaguNET.com>
References: <F3D7A615-23EF-4809-9BC3-9E1CA44212A5@jaguNET.com>
 <7C8BB910-7D19-4520-BE26-EE25CDC9608F@jaguNET.com>
 <CALK=YjNxnUAjX1mGsAXAeXRmHJ3Pz=bfcMnibKTc5j2Yf3QOSw@mail.gmail.com>
 <1012F319-C047-4527-8B01-903CDD0F083D@jaguNET.com>
 <56907444.2070905@primary.net>
To: dev@httpd.apache.org

Here is the general concept:

The health check itself is done via mod_watchdog, and uses
its callback impl. Each worker will have its own check interval,
which can be changed via the new balancer member directives
(hcinterval=3D, hcfails=3D,...); these will also be changable via
the balancer manager.

There will be a set of health checks available, depending on
the scheme of the backend. There will be:

  None
  TCP: a simple 'is socket up check'
  OPTIONS: for http only
  HEAD: http only, sends HEAD /...
  GET: http only, sends GET /...
  CPING: ajp cping/cpong

For the checks that send a URI, we allow for a specific
check URI (hcuri) that gets appended to the worker's url.
We will also read in the response, headers and all, and
create a dummy request_rec from it. Why? That way we can
create 'conditions' which use ap_expr to check for things
like REQUEST_STATUS, etc. These conditions will be
basically stored ap_expr expressions (ala <If>) that will
be applied to the returned response from the backend. Thus,
we can check not only that the backend is "up" (ie, it
responded), but also check the response itself.

These would be one-off requests, connecting directly to
the backend via ap_proxy_acquire_connection()/
ap_proxy_connect_backend()/ap_proxy_connection_create() with
the *dummy* generic reverse proxy worker (instead of the
actual backend worker itself, if you grok). It may make sense
to in addition to having the generic forward and reverse proxy
workers, also have a hcheck one as well, specialized for
this purpose.

I like the idea of parallel requests, and could likely impl
that as a simple thread pool in the watchdog process itself,
but that might be v1.1 or v1.2; I want to get the basic flow
itself right, esp the logic checking. So v1.0 will be serial. :/

> On Jan 8, 2016, at 9:45 PM, Daniel Ruggeri <DRuggeri@primary.net> =
wrote:
>=20
> ++1 on this vein of thought. Implementing the health checking similar =
to
> the modularized lb methods makes the most sense to me. I can already
> think of several methods (file, conf, external script, lua script) of
> obtaining data on how to probe and react.
>=20
> Just some thoughts worth considering... since you asked :-)
> *The module should not probe if a probe is already running against =
that
> backend (hang or takes too long to respond)
> *The probes to different backends should be done in parallel rather =
than
> serially to avoid pileups due to a slow responder
> *How will the existing balancer re-enable logic interfere with ERR =
backends?
>   *Maybe another flag that avoids retry if the monitor marked it down
> requiring the monitor to bring it back up before any traffic is =
allowed
> to go to it?
>=20
> I've followed along with what you described by using watchdog to =
trigger
> the probing, but I'm curious where your thoughts are for submitting a
> request to the backend. Are you planning to simulate one from your
> module by directly connecting to the backend or is the plan to run the
> request through the existing proxy infrastructure (the former allows
> lots of control while the latter permits other directives to come into
> play that might be handy like disabling based on status code). I =
haven't
> seen the code, but your previous email said you were thinking of the
> former case.
>=20
> P.S.
> Thanks for taking this on. It's been on my own todo list for a long =
time.
>=20
> --=20
> Daniel Ruggeri
>=20
> On 1/8/2016 1:09 PM, Jim Jagielski wrote:
>> Thx! Any other comments/suggestions?
>>=20
>>> On Jan 8, 2016, at 10:53 AM, Eric Covener <covener@gmail.com> wrote:
>>>=20
>>> On Fri, Jan 8, 2016 at 10:18 AM, Jim Jagielski <jim@jagunet.com> =
wrote:
>>>> I am thinking about the actual health-check impl a bit more,
>>>> hence the lack of commits. As noted, initially I was thinking
>>>> about optional functions, defined in mod_proxy_http, mod_proxy_ajp,
>>>> etc. Then I started thinking that maybe doing it via providers
>>>> might be better, allowing for more "custom" health checks via
>>>> the end user. Then I started thinking about doing it rough-and-
>>>> ready and just adding 'em in mod_proxy_hcheck...
>>>>=20
>>>> Before I spend too much time on this, any prefs one way or
>>>> another regarding this?
>>> I haven't looked at the architecture so far -- but I think basics in
>>> mod_proxy_hcheck
>>> and at least one "standalone" provider (as an example) would be a =
good
>>> middle ground.
>>>=20
>>> Maybe the standalone one would try to deal with the body whereas the
>>> built-in ones
>>> would be TCP up, SSL up, status-code, etc.
>=20