httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul J. Reder" <>
Subject Re: plz vote on tagging current CVS as APACHE_2_0_19
Date Fri, 22 Jun 2001 19:04:29 GMT wrote:
> > I don't completely understand you comment, but you are quite right that anytime
> > scoreboard needs to be walked, locking will be required. The scoreboard will -not-
need to
> > be walked except in these relatively rare cases:
> >
> > 1. During a graceful restart (manipulates elements in the scoreboard)
> > 2. During MaxRequestPerChild triggered child shutdown (manipulates elements in the
> > scoreboard)
> > 3. Processing a mod_status request (walks the scoreboard)
> > 4. Processing an SNMP request (walks the scoreboard)
> >
> > Requests that do not require a scoreboard walk or adding/deleting entries from the
> > scoreboard (which is 99.999% or more of all requests; all requests except the ones
> > will not require locking at all.  Let me restate this... mainline requests will
NOT be
> > required to make any lock calls at all. Ever.  Mainline requests can be served and
> > status tracked even when the scoreboard is locked, so this design does not suffer
> > performance penalties in the mainline case caused by locking.
> You are looking at this from a Web server point of view.  Don't.  Look at
> this from an ops point of view.  How often is a large site going to query
> the current connections or child status (for example) using SNMP?  Both of
> those require a walk of the scoreboard.  If you ask for them once a
> minute, you are going to seriously hinder the performance of your server.
> The problem gets worse as the number of threads increases.  How many other
> SNMP requests carry the same penalty?

Please. Locking the scoreboard once per minute for the period of a walk would
hardly impact performance. If locking the whole scoreboard is a problem, then
locking can be performed at finer granularities. You could lock the list of
processes, each list of workers, and each free list. Now you can have multiple
walks happening at the same time in different parts of the tree. The initial
version will have a simple lock of the whole scoreboard. *If* testing shows
this to be a problem, which I doubt, then we can look at finer granularity locking.

> BTW, there are a few more that modules that make use of the scoreboard
> information:
> 5.  Whenever Apache::Scoreboard walks the scoreboard (this has been used
> by perl modules to download the scoreboard once every 15 seconds to
> generate graphs on another machine.)

Once every 15 seconds is an eternity.

> 6.  Whenever Apache::VMonitor walks the scoreboard  (Another graphicl
> monitor of configurable refresh rate.
> 7.  Whenever any other management module walks the scoreboard.

People are used to tuning the access to get the info that they need without
impacting performance. It is a trade off. I'm sure there are those who would
like to have realtime statistics, but they recognize that isn't possible.

> You can't guarantee how often a third party module will walk the
> scoreboard, because you don't know everything about every third part
> module.  Since you don't know how often it will be walked, you don't know
> how often the mutex will be locked.

You are quite correct. But again, no one is going to install dozens of
statistics generating modules. No one is going to schedule statistics reporting
so often that it impacts service.

If someone sets the interval to be once per second, they will pay a price no
matter what scoreboard scheme is used. More often than that they pay a heavier
price. This isn't rocket science.

We have a tradeoff here. They may not be able to gather stats quite as often
due to the locking. The benefit that they get is a great deal more robustness
and flexibility. If they get a system that can scale better, perform better on
multi-processor machines, run faster and longer, and is easier to configure
do you think they might be willing to poll for statistics once every 5 seconds
instead of once per second?

And this is all handwaving anyway. You disagree. Fine. The true test will be the
performance of the end product.

> > The only reason we have locking at all is to prevent the 4 cases listed above from
> > colliding with each other.  Even in the 4 cases above, the lock contention will
be minimal
> > and the performance degradation minimal and perhaps not even measureable.
> >
> > A few other benefits to Pauls design:
> > 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
> You still need to request a certain amount of shared memory when you
> start the parent process, and you will require HARD_SERVER_LIMIT and
> HARD_THREAD_LIMIT to know how much to request.  Either that, or whenever
> you add a new child, you will need to allocate more shared memory, and
> somehow tell your child processes about it.

The allocation is based on the configured values for the MaxClients and ThreadsPerChild.
These values can changed from one start to the next (allowing system tuning to happen
between restarts - without a recompile!)

> > 2. You don't need to allocate child score if you don't care about mod_status (but
it can
> > be added during a restart)
> You still need to request a certain amount of shared memory when you
> start the parent process, and you will require HARD_SERVER_LIMIT and
> HARD_THREAD_LIMIT to know how much to request.  Either that, or whenever
> you restart, you will forget all information about your child processes.

See above for allocation size basis. As for keeping info from one restart to the next,
the code currently allocates a new scoreboard on graceless restarts and all info from the
old scoreboard is lost. On gracefull restarts the existing scoreoard is reused. In the
future gracefull restarts would get a clean scorebaord and the old one would eventually
be cleaned up. The results would be collected and stored at a common top level.
> > 3. If you choose to not enable mod_status, you will likely see a nice performance
boost on
> > multi CPU machines because we are not invalidating a CPUs cache each time we touch
> > worker score entry (which is on every request).
> This can be implemented just as well with the table implementation.  The
> only thing you have to do is pad the scoreboard entry size to make it
> equal to one cache line.

Padding the entries uses more memory but makes mapping into the cache easier. It *does not*
in any way limit the cache invalidation impact of updating scoreboard entries *that are never
read* when a status module is not loaded.

> > 4. Does not require any changes to the MPM.  Each MPM can start threads according
to its'
> > ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe
> > design required child processes to wait for worker score slots to become available
> > it can start threads. This is imposing too much unnecessary work on the MPM.).
> You are ignoring one problem with the design.  I have now posted about it
> three times, and nobody has told me how it will be fixed.  You need to
> honor MaxClients.  If I set MaxClients to 250, then I expect MaxClients to
> be 250, regardless of whether those clients are long-lived or not.  If
> every child can always start threads according to ThreadsPerChild, you
> will be violating MaxClients on a heavily loaded server.  This means that
> you are actually requiring MORE changes to the starting logic than the
> table implementation, because each child, will need to determine how many
> threads to start at any given time.

There is currently code to look if another process can be started (in 
process_idle_server_maintenance), there will still be code to do that with
my design. The design still limits the number of processes and the number
of workers per process. The difference is that a new process can be started
when (current_total_workers < ((MaxClients-1) * ThreadsPerChild)) provided
there is a spare process.

"My design can handle this" (I heard you say that all the way from CA). This
is true to a point. Currently, in the table design, a process cannot own
workers from mutiple rows. A process owns all of the workers in a given row.
Currently there is a one to one mapping of process slots to worker rows.
Your proposal is to allow multiple processes to map to a single row, but still
not map a process to workers in multiple rows.

Apache does a wonderful job of distributing the request load to workers. 
There is a cycle that happens where many workers start running into the MRPC
limit and start quiescing. As workers quiesce you will see 5 freed up from
this row, and 6 from that, and another 10 from that...

You will need to be able to start up a new process per row to take advantage
of these dribs and drabs that are appearing. With my design, as soon as there
is ThreadsPerChild number of workers on the free list, and there is a free
process, I can start a new process with ThreadsPerChild workers.

What am I missing here? There is no overrun of the configured values. And the
algorithm isn't any more complex.

Paul J. Reder
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it.  Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein

View raw message