Many thanks for all the answers on this one folks - in summary the 
suggestions were packet tracing, mod_security, RewriteLog, and 
mod_log_forensic.

I went for packet tracing using tshark with a capture condition to 
minimise the performance impact, which worked nicely.  The requests in 
question were completely missing, so the next possibility is an 
occasional DNS lookup failure.

cheers
John

Paul Querna wrote:
>
> mod_log_forensic was created for exactly this purpose:
> <http://httpd.apache.org/docs/2.2/mod/mod_log_forensic.html>
>
> hope that helps,
>
> Paul
>
>   

On Thu, Dec 3, 2009 at 12:43 AM, John ORourke <john-perl@o-rourke.org> wrote:
> (apologies if this is a dupe, I originally sent from the wrong address)
>
> Hi there,
>
> I have an unusual problem - a large e-commerce site integrated with
> Authorize.net for card payments which appears to be failing to log some
> requests.
>
> The Authorize.net system makes HTTP POST requests to our server, and
> about 1 in every 500 transactions, the Authorize.net system reports a
> timeout and there's no trace of the request in our logs.  Authorize.net
> won't investigate in any detail because their system is reporting that
> the request simply timed out.
>
> I'm using mod_perl but not hooking into the logging phase, and using a
> mod_log CustomLog directive which outputs the usual stuff, plus request
> time, connection state, and PID.
>
> So for now, I'm assuming a request is being sent but the log handler
> phase isn't running.  The only way I can make this happen in a test
> environment is by opening a TCP connection and then closing it without
> sending any data.  Are there any other reasons the log phase wouldn't be
> run?

mod_log_forensic was created for exactly this purpose:
<http://httpd.apache.org/docs/2.2/mod/mod_log_forensic.html>

hope that helps,

Paul

Niklas Edmundsson wrote:
> On Mon, 30 Nov 2009, William A. Rowe Jr. wrote:
> 
>> Look, PCRE is a mandatory component.  APR is a mandatory component.
>> Let's please start applying some rhyme to our reasoning again.
> 
> +1

+1 a good write up.


-- 
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70  3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354
VP Apache Infrastructure; Member, Apache Software Foundation
Committer,                        FreeBSD Foundation
Consultant,                       P6M7G8 Inc.
Sr. System Admin,                 Ridecharge Inc.

Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.

Joe Orton wrote:
> On Thu, Dec 03, 2009 at 05:21:09PM -0600, William Rowe wrote:
>> Paul Querna wrote:
>>> Vote Results:
>>>    +1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
>>>    +1: Gregg Smith
>>>  +/-0: Rainer Jung
>>>     -1: William A. Rowe, Jr.
>>>
>>> Vote passes.
>> I'm sorry.  I explicitly insisted on a vote on the -deps package seperately
>> from the 2.3.4 package, because it was entirely reasonable that Sander Temme,
>> Paul Querna, Joe Orton, Niklas Edmundsson, or Gregg Smith reviewed -only- the
>> httpd-2.3.4-alpha.tar.xx package alone.
> 
> I've no issue at all with the -deps tarball containing a snapshot of 
> APR:
> 
> 1) the httpd project cannot force the APR project to commit to API 
> stability by distributing a snapshot of the APR 1.4 branch.  Why on 
> earth would that be the case?  The only time the APR project commits to 
> API stability is by making a new .0 release itself.  What other projects 
> do is irrelevant to APR.

Joe, as I pointed out in another thread, httpd *does* commit apr the moment
the mislabeled artifact hits www.apache.org/dist/ - where does that package
suggest it is nothing but a snapshot?  The answer is, it doesn't.

When we did this previously in httpd 2.0 (never 2.1 to the best of my
recollection) we actually did commit to that particular apr API.  The
rules in 0.9 just weren't so draconian.

If you don't like any of these rules;

 * what is on /dist/ is a release
 * what is released apr >= 1.0 follows absolute versioning rules
 * what is a snapshot doesn't appear on /dist/

then take those issues to an infra/board level.  We've had these discussions
a thousand times on incubator lists, and this is a clear case of what is
good for the goose...

> 2) the httpd project isn't taking on any commitment to itself maintain 
> API stability in the shipped APR snapshot because *this is an alpha*, so 
> we're not guaranteeing API stability.

Nope, much like broken ldap and crypto interfaces, it simply inflicts them
upon apr and hopes for someone else to clean up the junk later, perhaps.

But the apr project would be remiss in not voiding 1.4.x and moving on to
1.5.x for API additions since users can now be reasonably expected to have
installed an apr 1.4.x major/minor with a specific feature set.

> Furthermore I don't think it's a good idea to set a precedent of 
> requiring a separate vote on each file which makes up "the release".  I 
> certainly presumed that the vote Paul called was for all the files 
> making up "the release".

Nonsense.  I post up httpd win32 binaries all the time to /dev/dist/.  Do
I really think everyone voting on the source code snapshot of httpd is paying
any attention to that artifact (or this new -deps artifact, when all the deps
are already provisioned on most voter's machines)?

Paul Querna wrote:
> Vote Results:
>    +1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
>    +1: Gregg Smith
>  +/-0: Rainer Jung
>     -1: William A. Rowe, Jr.
> 
> Vote passes.
> 
> I'll push out the tarballs to start getting mirrors, and hack on an
> announcement email for tomorrow.

Oh - looking forward to the announce; the damage occured the moment
you had pushed to /dist/httpd/, we don't un-release software, and any
old arguments are over about the wisdom of releasing the -deps package.
Except of course, we need to finish the discussion of *adding* PCRE, but
that should never stop an alpha release.

In the meantime I'm pushing forwards on apr-1.4.1 since testing for
subversion 0 won't indicate if the APR_STATUS_IS_ENOTDIR had been fixed
(for anyone looking for min revlevel checks), and once that is done,
perhaps someone else wants to advocate for releasing apr-util-1.4.0.
But AFAIK apr-util 1.4.0 is not a prerequisite for httpd 2.3 alphas
just yet, it is an optional requirement for the scache module.

Jeff Trawick wrote:
> On Thu, Dec 3, 2009 at 11:29 PM, William A. Rowe Jr.
> 
>> As for broken versioning rules, please take that to APR.
>>
>> Perhaps in retrospect, APR would consider an even/odds approach as httpd
>> has for adding (even eliminating) interfaces during a development cycle.
> 
> IMO the determination could be as simple as whether or not a release
> in the maj.min series has yet been declared GA.

Not sufficient, IMHO, unless we add the appropriate indicators to allow
feature tests in apr 2.0.  E.g. the current test is VERSION_MAJOR >= 1
&& VERSION_MINOR >= 4 and the APR versioning contract says that test is
always sufficient.

This dialog reads like folks may assume the httpd MMN feature tests
work for APR functions, but there isn't such a feature test in APR.

Jeff Trawick wrote:
> On Thu, Dec 3, 2009 at 11:29 PM, William A. Rowe Jr.
> <wrowe@rowe-clan.net> wrote:
>> Jeff Trawick wrote:
>>> On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr. <wrowe@rowe-clan.net>
wrote:
>>>
>>>> Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
>>>> apr-util 1.4.0 dev, and the API versioning rules will bind from that release
>>>> forwards.
>>> The APR versioning rules are hopelessly broken if a tarball snapshot
>>> of the 1.4.x branch before a GA release casts the API in stone.
>>>
>>> Surely I misunderstood you.
>> Is there a README indicating that the MAJOR/MINOR version tests for this
>> particular tarball are not relevant/complete?  No.
>>
>> This is not a snapshot.  It is labeled httpd-2.3.4-alpha.tar.xx release.
>> You surely don't misunderstand what I said.
> 
> Why is something with version x.y.z-dev a release and not a snapshot?

Because snapshots don't live at http://www.apache.org/dist/, those are releases.
The trigger didn't occur until Paul svn mv'ed it into there.  Snapshots reside
at http://svn.apache.org/snapshots/

>> As for broken versioning rules, please take that to APR.
>>
>> Perhaps in retrospect, APR would consider an even/odds approach as httpd
>> has for adding (even eliminating) interfaces during a development cycle.
> 
> IMO the determination could be as simple as whether or not a release
> in the maj.min series has yet been declared GA.

Eric Covener wrote:
> On 12/4/09, Barry Scott <barry.scott@onelan.co.uk> wrote:
>   
>> Eric Covener wrote:
>>
>>     
>>> On Thu, Dec 3, 2009 at 5:57 AM, Barry Scott <barry.scott@onelan.co.uk>
>>>       
>> wrote:
>>     
>>>       
>>>> Jeff Trawick wrote:
>>>>
>>>>
>>>>         
>>>>> On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank <ef-lists@email.de>
>>>>>           
>> wrote:
>>     
>>>>> In the interim, is mod_fastcgi really that bad?
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> mod_fastcgi is fine for handling GET/POST requests, but it fails to
>>>> implement
>>>> Authorization or Authenication.
>>>>
>>>> So yes mod_fastcgi is really bad.
>>>>
>>>>
>>>>         
>>> mod_fastcgi has supported this for many years:
>>>
>>> http://www.fastcgi.com/drupal/node/25#FastCgiAuthorizer
>>>
>>>       
>> http://www.fastcgi.com/drupal/node/25#FastCgiAuthenticator
>>     
>>>
>>>       
>>  It does not work or I'd have used it. And I tried to make it work.
>>  There is a lot of missing code, compare to mod_fcgid implementation
>>  of the same.
>>     
>
> Simple tests work for me.
>
>   

Hmm, Then I must have got something wrong when I tried to get this 
working, or you have
patches I don't. When I looked at the sources and compared to mod_fcgid 
it looked like there
was code missing. It's too long ago now for me to recall the details to 
discuss.

Barry

On 12/4/09, Tom Evans <tevans.uk@googlemail.com> wrote:
> Sorry, what has apachectl got to do with editing files? What has using
>  apachectl to stop/start a service got to do with scalability? You've
>  completely lost me here.

The only practical thing i can think of is OS vendors providing
separate worker and prefork binaries. The standard apachectl has the
binary name embedded in it, although there are a host of ways to
accomodate that.

-- 
Eric Covener
covener@gmail.com

On Fri, Dec 4, 2009 at 12:37 PM, Graham Leggett <minfrin@sharp.fm> wrote:
> Tom Evans wrote:
>
>> Really? It works perfectly on all boxes I use it on. What precisely
>> has changed about reading a pid from a file, sending signals to a
>> process, or spawning a process with specific arguments that has made
>> apachectl 'archaic and largely broken', I am intrigued.
>
> And if you have ten boxes? 50 boxes? A Google of boxes?
>
> Editing a file in place has long been shown to be a maintenance
> nightmare, which is why in addition to logrotate.conf you have
> logrotate.d, in addition to modprobe.conf you have modprobe.d, and so
> on. This is a pattern long since established in modern unix
> distributions, to solve the problem of the need to edit files during a
> software addition, and edit files again during software removal, all
> without making mistakes.
>
> Sure, if you are used to editing config files by hand on one or two
> boxes, apachectl will meet your needs, but if you do anything that
> requires a level of scale doing it this way won't.
>
> Regards,
> Graham
> --
>

Sorry, what has apachectl got to do with editing files? What has using
apachectl to stop/start a service got to do with scalability? You've
completely lost me here.

At $JOB we have 200+ servers, all deployed and configured via
cfengine. apachectl is still used to stop/start the servers, via
cfengine and other CM tools.

Cheers

Tom

On 12/4/09, Barry Scott <barry.scott@onelan.co.uk> wrote:
> Eric Covener wrote:
>
> > On Thu, Dec 3, 2009 at 5:57 AM, Barry Scott <barry.scott@onelan.co.uk>
> wrote:
> >
> >
> > > Jeff Trawick wrote:
> > >
> > >
> > > > On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank <ef-lists@email.de>
> wrote:
> > > >
> > > > In the interim, is mod_fastcgi really that bad?
> > > >
> > > >
> > > >
> > > mod_fastcgi is fine for handling GET/POST requests, but it fails to
> > > implement
> > > Authorization or Authenication.
> > >
> > > So yes mod_fastcgi is really bad.
> > >
> > >
> >
> > mod_fastcgi has supported this for many years:
> >
> > http://www.fastcgi.com/drupal/node/25#FastCgiAuthorizer
> >
> http://www.fastcgi.com/drupal/node/25#FastCgiAuthenticator
> >
> >
> >
>  It does not work or I'd have used it. And I tried to make it work.
>  There is a lot of missing code, compare to mod_fcgid implementation
>  of the same.

Simple tests work for me.

-- 
Eric Covener
covener@gmail.com

Tom Evans wrote:

> Really? It works perfectly on all boxes I use it on. What precisely
> has changed about reading a pid from a file, sending signals to a
> process, or spawning a process with specific arguments that has made
> apachectl 'archaic and largely broken', I am intrigued.

And if you have ten boxes? 50 boxes? A Google of boxes?

Editing a file in place has long been shown to be a maintenance
nightmare, which is why in addition to logrotate.conf you have
logrotate.d, in addition to modprobe.conf you have modprobe.d, and so
on. This is a pattern long since established in modern unix
distributions, to solve the problem of the need to edit files during a
software addition, and edit files again during software removal, all
without making mistakes.

Sure, if you are used to editing config files by hand on one or two
boxes, apachectl will meet your needs, but if you do anything that
requires a level of scale doing it this way won't.

Regards,
Graham
--

On Fri, Dec 4, 2009 at 12:59 AM, Graham Leggett <minfrin@sharp.fm> wrote:
> William A. Rowe Jr. wrote:
>
>> Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
>> does this say for using our httpd rpm for an Ubuntu or other distribution
>> of linux?
>
> Ubuntu is Debian based, and uses the .deb packaging format, and startup
> scripts derived from the Debian layout. If someone was to donate debian
> packaging for httpd, I would expect one or two files to appear below
> build/deb, and that would be all that would be needed.
>
>> IMHO, if it is fundamentally incompatible with apachectl and non-redhat
>> distributions, let the the packagers tweak and take the zany customizations
>> out from under our problem set.
>
> Apachectl is archaic and largely broken for most people - it made sense
> ten years ago, it makes a lot less sense today.

Really? It works perfectly on all boxes I use it on. What precisely
has changed about reading a pid from a file, sending signals to a
process, or spawning a process with specific arguments that has made
apachectl 'archaic and largely broken', I am intrigued.


>
> The pattern followed by most modern unix based packaging is for an
> application to drop a snippet of config contained in a discrete file in
> a directory ending in ".d". So you have
> /etc/httpd/conf.d/<snippet>.conf, instead of a manual edit to
> /etc/httpd/conf/httpd.conf, and your httpd startup goes within an
> optional script called /etc/sysconfig/httpd instead of in a script file
> in a bin directory as apachectl wants. I understand Debian has different
> naming conventions, but otherwise the underlying principles are the same.

Did you mean to say 'most Linux' there? The OSes that I regularly use
do not display these redhatisms.

>
> In our case, we package up config files within standalone RPMs all of
> their own (suitably tagged and versioned), or we generate the config
> file using puppet. Editing a file in place is always painful and error
> prone, it is far less painful to provide a discrete file that can be
> dropped in and removed cleanly. Apachectl doesn't give us this - you
> need to edit apachectl directly to modify the command line parameters
> passed to httpd.
>
> As for the packagers tweaking and making zany customisations, that is
> exactly what they do now. For us it makes the their packaging unsuitable
> for our needs, simply because we tweak and make zany customisations for
> needs of our own. It is far less painful to take a vanilla RPM published
> by the ASF, and then tweak it for our needs, than it is to take a Fedora
> RPM, untweak all their customisations, and then retweak it with ours.
>

Ah so now the crux of your argument:

1) I use Fedora/RHEL
2) I want customized packages to install
3) Fedora/RHEL package their RPMs in such a way that it makes it
difficult for me to modify them.

It's much easier when you just say this up front.

Cheers

Tom

On Thu, Dec 3, 2009 at 11:29 PM, William A. Rowe Jr.
<wrowe@rowe-clan.net> wrote:
> Jeff Trawick wrote:
>> On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr. <wrowe@rowe-clan.net> wrote:
>>
>>> Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
>>> apr-util 1.4.0 dev, and the API versioning rules will bind from that release
>>> forwards.
>>
>> The APR versioning rules are hopelessly broken if a tarball snapshot
>> of the 1.4.x branch before a GA release casts the API in stone.
>>
>> Surely I misunderstood you.
>
> Is there a README indicating that the MAJOR/MINOR version tests for this
> particular tarball are not relevant/complete?  No.
>
> This is not a snapshot.  It is labeled httpd-2.3.4-alpha.tar.xx release.
> You surely don't misunderstand what I said.

Why is something with version x.y.z-dev a release and not a snapshot?

> As for broken versioning rules, please take that to APR.
>
> Perhaps in retrospect, APR would consider an even/odds approach as httpd
> has for adding (even eliminating) interfaces during a development cycle.

IMO the determination could be as simple as whether or not a release
in the maj.min series has yet been declared GA.

William A. Rowe Jr. wrote:

> The last I heard, the 'rpm' project is open source, free to be adopted by
> any platform.

Just because rpm is free to be adopted by any platform doesn't mean it
has been. Rpm contains features that allow the spec file to tailor
itself to its build environment, and I am confident those features would
kick in if someone was to try and create an rpm package tailored for an
environment other than a Redhat derivative.

> As for the rest of your comments, if we solve the general problem, I'm all
> for including it in the httpd tree.  If we are solving specific packagers
> problems, I'm ok with placing this in the httpd.a.o domain, but we should
> move this nonsense outside of the development tree into a packaging tree.

And in turn violate the principle of least surprise.

The point behind support for packaging formats is to make the end user's
life easier, not harder. Packagers that create weird, non standard, or
unexpectedly complex packaging are not doing their users any favours.

Regards,
Graham
--

Eric Covener wrote:
> On Thu, Dec 3, 2009 at 5:57 AM, Barry Scott <barry.scott@onelan.co.uk> wrote:
>   
>> Jeff Trawick wrote:
>>     
>>> On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank <ef-lists@email.de> wrote:
>>>
>>> In the interim, is mod_fastcgi really that bad?
>>>
>>>       
>> mod_fastcgi is fine for handling GET/POST requests, but it fails to
>> implement
>> Authorization or Authenication.
>>
>> So yes mod_fastcgi is really bad.
>>     
>
> mod_fastcgi has supported this for many years:
>
> http://www.fastcgi.com/drupal/node/25#FastCgiAuthorizer
> http://www.fastcgi.com/drupal/node/25#FastCgiAuthenticator
>
>   
It does not work or I'd have used it. And I tried to make it work.
There is a lot of missing code, compare to mod_fcgid implementation
of the same.

Barry

On Thu, Dec 03, 2009 at 05:21:09PM -0600, William Rowe wrote:
> Paul Querna wrote:
> > Vote Results:
> >    +1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
> >    +1: Gregg Smith
> >  +/-0: Rainer Jung
> >     -1: William A. Rowe, Jr.
> > 
> > Vote passes.
> 
> I'm sorry.  I explicitly insisted on a vote on the -deps package seperately
> from the 2.3.4 package, because it was entirely reasonable that Sander Temme,
> Paul Querna, Joe Orton, Niklas Edmundsson, or Gregg Smith reviewed -only- the
> httpd-2.3.4-alpha.tar.xx package alone.

I've no issue at all with the -deps tarball containing a snapshot of 
APR:

1) the httpd project cannot force the APR project to commit to API 
stability by distributing a snapshot of the APR 1.4 branch.  Why on 
earth would that be the case?  The only time the APR project commits to 
API stability is by making a new .0 release itself.  What other projects 
do is irrelevant to APR.

2) the httpd project isn't taking on any commitment to itself maintain 
API stability in the shipped APR snapshot because *this is an alpha*, so 
we're not guaranteeing API stability.

Furthermore I don't think it's a good idea to set a precedent of 
requiring a separate vote on each file which makes up "the release".  I 
certainly presumed that the vote Paul called was for all the files 
making up "the release".

Regards, Joe

Graham Leggett wrote:
> William A. Rowe Jr. wrote:
> 
>> Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
>> does this say for using our httpd rpm for an Ubuntu or other distribution
>> of linux?
> 
> Ubuntu is Debian based, and uses the .deb packaging format, and startup
> scripts derived from the Debian layout.

The last I heard, the 'rpm' project is open source, free to be adopted by
any platform.

As for the rest of your comments, if we solve the general problem, I'm all
for including it in the httpd tree.  If we are solving specific packagers
problems, I'm ok with placing this in the httpd.a.o domain, but we should
move this nonsense outside of the development tree into a packaging tree.

Jeff Trawick wrote:
> On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr. <wrowe@rowe-clan.net> wrote:
> 
>> Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
>> apr-util 1.4.0 dev, and the API versioning rules will bind from that release
>> forwards.
> 
> The APR versioning rules are hopelessly broken if a tarball snapshot
> of the 1.4.x branch before a GA release casts the API in stone.
> 
> Surely I misunderstood you.

Is there a README indicating that the MAJOR/MINOR version tests for this
particular tarball are not relevant/complete?  No.

This is not a snapshot.  It is labeled httpd-2.3.4-alpha.tar.xx release.
You surely don't misunderstand what I said.

As for broken versioning rules, please take that to APR.

Perhaps in retrospect, APR would consider an even/odds approach as httpd
has for adding (even eliminating) interfaces during a development cycle.

William A. Rowe Jr. wrote:
>> I may be wrong but as an outsider looking in, I see you wanting to stop maintaining/including
the gear box and are instead spending the time on adding more optional gadgets to choose from
(some of the third party modules you've taken over). In the end, I'd prefer having a reverse
gear over the rear window defogger. You are also loosing all control of a required piece of
equipment, this has got to make some of you at least cringe a little.
> 
> I'm not 100% sure I understand what you are saying here.  Drop the
> gearbox and let them assemble their own transmission?  Or distribute
> a most modern transmission that the user can ignore or swap out if they
> want to install their own?
> 

Bill,

The latter in your statement, distribute the most modern transmission 
that the user can ignore or if they wish or swap out with something they 
prefer.

Basically it's easy. If it must be there to build the darn thing, pass 
GO and collect the $$$, then it should be there. If not, then it becomes 
an option and that is left up to the user/builder ergo openssl zlib and 
lua.

so +1 to including pcre in -deps (or srclib/pcre), basically as it is 
now in the other branches.

In reality I probably shouldn't say anything anymore as I adapted and 
overcame, begrudgingly. It sure was a sore subject back between 2.3.1 
and 2.3.2 and I guess it just held on and I could not resist making noise.

Sorry this got into the wrong thread, I blame my webmail :)

Gregg

William A. Rowe Jr. wrote:

> Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
> does this say for using our httpd rpm for an Ubuntu or other distribution
> of linux?

Ubuntu is Debian based, and uses the .deb packaging format, and startup
scripts derived from the Debian layout. If someone was to donate debian
packaging for httpd, I would expect one or two files to appear below
build/deb, and that would be all that would be needed.

> IMHO, if it is fundamentally incompatible with apachectl and non-redhat
> distributions, let the the packagers tweak and take the zany customizations
> out from under our problem set.

Apachectl is archaic and largely broken for most people - it made sense
ten years ago, it makes a lot less sense today.

The pattern followed by most modern unix based packaging is for an
application to drop a snippet of config contained in a discrete file in
a directory ending in ".d". So you have
/etc/httpd/conf.d/<snippet>.conf, instead of a manual edit to
/etc/httpd/conf/httpd.conf, and your httpd startup goes within an
optional script called /etc/sysconfig/httpd instead of in a script file
in a bin directory as apachectl wants. I understand Debian has different
naming conventions, but otherwise the underlying principles are the same.

In our case, we package up config files within standalone RPMs all of
their own (suitably tagged and versioned), or we generate the config
file using puppet. Editing a file in place is always painful and error
prone, it is far less painful to provide a discrete file that can be
dropped in and removed cleanly. Apachectl doesn't give us this - you
need to edit apachectl directly to modify the command line parameters
passed to httpd.

As for the packagers tweaking and making zany customisations, that is
exactly what they do now. For us it makes the their packaging unsuitable
for our needs, simply because we tweak and make zany customisations for
needs of our own. It is far less painful to take a vanilla RPM published
by the ASF, and then tweak it for our needs, than it is to take a Fedora
RPM, untweak all their customisations, and then retweak it with ours.

Regards,
Graham
--

On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr. <wrowe@rowe-clan.net> wrote:

> Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
> apr-util 1.4.0 dev, and the API versioning rules will bind from that release
> forwards.

The APR versioning rules are hopelessly broken if a tarball snapshot
of the 1.4.x branch before a GA release casts the API in stone.

Surely I misunderstood you.

Paul Querna wrote:
> 
> I don't agree that we can't release a bundled unreleased version of
> APR, we did this for many versions of httpd 2.0.x and 2.1.x.  It
> definitely isn't preferred, but that's the APR project's problem.

Look, your argument simply doesn't fly.

In httpd 2.0 timeframe we were only shipping apr-0.9.x - it did NOT
have the same API/ABI constraints (some of them, but not all).  All
of those intermediary releases kept the ABI rules of APR.

Now that you have shipped immediately while ignoring my objection,
I'll treat all +1's as binding no matter if they approved both of
the pieces or not, and have tagged 1.4.0 of both apr and apr-util.

We have no alternative, or else all author's VERSION_MAJOR/MINOR
tests are invalid.

It becomes up to the APR project if this aught to be 1.4.0 or burn
a number and move on.  For 1.4 initial release, I want to pick up
Branko's fix, so I plan to label this 1.4.1.

No intention of tagging apr-util yet till we decide if it can be
API frozen, so if we dislike the current includes/ tree, it will
end up being deprecated interfaces and version 1.5.0 already.

Paul Querna wrote:
> Vote Results:
>    +1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
>    +1: Gregg Smith
>  +/-0: Rainer Jung
>     -1: William A. Rowe, Jr.
> 
> Vote passes.

I'm sorry.  I explicitly insisted on a vote on the -deps package seperately
from the 2.3.4 package, because it was entirely reasonable that Sander Temme,
Paul Querna, Joe Orton, Niklas Edmundsson, or Gregg Smith reviewed -only- the
httpd-2.3.4-alpha.tar.xx package alone.

Gentlemen, if you could verify that your vote was for *both* the core package
and the -deps.tar.xx package, that might keep Paul on his schedule (or fall
against the proposed -deps.tar.xx package).

Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
apr-util 1.4.0 dev, and the API versioning rules will bind from that release
forwards.

Vote Results:
   +1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
   +1: Gregg Smith
 +/-0: Rainer Jung
    -1: William A. Rowe, Jr.

Vote passes.

I'll push out the tarballs to start getting mirrors, and hack on an
announcement email for tomorrow.

On Wed, Nov 25, 2009 at 2:43 PM, Paul Querna <paul@querna.org> wrote:
> Test tarballs for Apache httpd 2.3.4-alpha are available at:
>   <http://httpd.apache.org/dev/dist/>
>
> Your votes please;
>
>  +/- 1
>  [  ]  Release httpd-2.3.4 as Alpha
>
> Vote closes at 18:00 UTC on Monday November 30 2009.
>
> May your Thanksgiving be filled with Turkey and httpd testing,
>
> Thanks,
>
> Paul
>

Gregg L. Smith wrote:
> Original Message -----------------------
>> Finally, I have yet to see any feedback on the pcre mandatory 
>> dependency issue.  Comments?
> 
> Personally, I thought your Monopoly metaphor was quite on target.
> 
> libz, openssl, lua = batteries not included
> apr, apu, pcre = drive train not included.
> 
>> And what is passing for an excuse for a local PCRE install 
>> these days probably doesn't look like 7.8 or later, with 
>> various fixes we are vulnerable to.
> 
> This does not leave me with a warm and fuzzy feeling. As a user, is the pcre 8.0 I've
built going to expose me to risks that your maintained 7.8 does not? If yes, then I'd prefer
your maintained one. After all, who knows better than you what will interact with your code
to produce problems. Regardless of merit, who will ultimately get blamed in the end? Could
your reputation be tarnished? Can you completely divorce yourself from something your software
requires to run?

I'm referring to pre v7 chaos.  And mostly not referring to modern
linux distros.

> The 'Jump Ship' factor;
> 
> To me, and I'm probably wrong, it sounds like Mr. Felt's comment was an ultimatum of
sorts as 'indefinitely' is a pretty strong word. With this issue you have created a deal with
it or jump ship ultimatum which could very well leave some people scrambling to get off. Each
person is going to inevitably weigh the pain factor, the pain of dealing with it over the
pain of jumping ship. I consider myself lucky that my second attempt to deal with it was successful,
or so it seems so far anyway, but I never know from day to day.

Agreed that ease-of-adoption is going to be the usual, first barrier to
anyone jumping aboard 2.4 from 2.2, 2.0, or even still from 1.3.

> I may be wrong but as an outsider looking in, I see you wanting to stop maintaining/including
the gear box and are instead spending the time on adding more optional gadgets to choose from
(some of the third party modules you've taken over). In the end, I'd prefer having a reverse
gear over the rear window defogger. You are also loosing all control of a required piece of
equipment, this has got to make some of you at least cringe a little.

I'm not 100% sure I understand what you are saying here.  Drop the
gearbox and let them assemble their own transmission?  Or distribute
a most modern transmission that the user can ignore or swap out if they
want to install their own?

> Sorry for the outburst, but you opened the door for, and I've said what I've wanted to
for some time now, thanks for listening. Corrections and daggers welcomed.

No problems, thanks for chiming in.

Graham Leggett wrote:
> 
> On a Redhat machine (Fedora/RHEL/Centos), search
> /etc/rc.d/init.d/functions for functions called "daemon", "status" and
> "killproc". These functions provide similar but incompatible
> functionality to that provided by apachectl, and only exist on Redhat
> derived machines.

Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
does this say for using our httpd rpm for an Ubuntu or other distribution
of linux?

> The startup script is far too trivial to justify jumping through hoops
> to try and make apachectl work like Redhat's init. It's caused us enough
> grief already, thus the fix.

IMHO, if it is fundamentally incompatible with apachectl and non-redhat
distributions, let the the packagers tweak and take the zany customizations
out from under our problem set.

On Thu, Dec 3, 2009 at 5:57 AM, Barry Scott <barry.scott@onelan.co.uk> wrote:
> Jeff Trawick wrote:
>>
>> On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank <ef-lists@email.de> wrote:
>>
>> In the interim, is mod_fastcgi really that bad?
>>
>
> mod_fastcgi is fine for handling GET/POST requests, but it fails to
> implement
> Authorization or Authenication.
>
> So yes mod_fastcgi is really bad.

mod_fastcgi has supported this for many years:

http://www.fastcgi.com/drupal/node/25#FastCgiAuthorizer
http://www.fastcgi.com/drupal/node/25#FastCgiAuthenticator

-- 
Eric Covener
covener@gmail.com

John ORourke <john-perl@o-rourke.org> writes:

> The Authorize.net system makes HTTP POST requests to our server, and
> about 1 in every 500 transactions, the Authorize.net system reports a
> timeout and there's no trace of the request in our logs.  Authorize.net
> won't investigate in any detail because their system is reporting that
> the request simply timed out.

This could happen in Apache 2 if the web server timed out before reading
the complete request.  (Apache 1.3 logs a 408; Apache 2 doesn't; Apache
trunk will log a 4xx of some sort depending on how much of the request
was read.)

If that's what's happening, it would indicate a problem outside of
Apache.  You could confirm it by backporting the fix from
https://issues.apache.org/bugzilla/show_bug.cgi?id=39785 so logging
would occur.  Then you might need to resort to packet tracing or
something like that to figure out exactly what's going on.

Dan

On 3 Dec 2009, at 08:43, John ORourke wrote:

> So for now, I'm assuming a request is being sent but the log handler
> phase isn't running.

Seems improbable, though mod_perl adds an extra layer.  Graham's
suggestion looks more plausible.

> My next steps are to add simple request logging during the Trans phase,

You can do that by adding a match-and-do-nothing RewriteRule and
setting RewriteLogLevel to log it.  Or the third-party mod_security
will log a great deal more for you.

btw, this should be on the users@ list.

-- 
Nick Kew

You might also take a look into global apache's error log. There are very
rare cases when PHP (or, probably, other modules) causes segmentation faults
in apache's child processes which prevents any request info to be written to
access log. The last time I had problems with this issue it was caused by
Oracle PHP extension (oci).

2009/12/3 John ORourke <john-perl@o-rourke.org>

> That gives me a few new places to hunt down the issue, thanks.
>

Regards,
Paul

Jeff Trawick wrote:
> On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank <ef-lists@email.de> wrote:
>   
>
> In the interim, is mod_fastcgi really that bad?
>
>   
mod_fastcgi is fine for handling GET/POST requests, but it fails to 
implement
Authorization or Authenication.

So yes mod_fastcgi is really bad.

mod_fcgid is a very welcome as  a supported httpd module.

Barry

Graham Leggett wrote:
> If httpd isn't logging anything, the most likely explanation is that the
> request isn't reaching httpd at all.
>
> How reliable is your network between their system and yours? Are there
> any load balancing devices or other network magic in the way that could
> potentially be misconfigured?
>   
That gives me a few new places to hunt down the issue, thanks.

> Packet sniffing will answer the question "was there any evidence of a
> request", and would probably be the least invasive way of measuring
> this. It's always useful to see what the network is actually doing,
> rather than what you think it's doing.
>   

Definitely - I've decided to bite the bullet and run tshark in the 
background, hopefully that will turn up some new information.  Debugging 
an error which happens at random a few times a month is an interesting 
challenge!

cheers
John

John ORourke wrote:

> I have an unusual problem - a large e-commerce site integrated with
> Authorize.net for card payments which appears to be failing to log some
> requests.
> 
> The Authorize.net system makes HTTP POST requests to our server, and
> about 1 in every 500 transactions, the Authorize.net system reports a
> timeout and there's no trace of the request in our logs.  Authorize.net
> won't investigate in any detail because their system is reporting that
> the request simply timed out.

If httpd isn't logging anything, the most likely explanation is that the
request isn't reaching httpd at all.

How reliable is your network between their system and yours? Are there
any load balancing devices or other network magic in the way that could
potentially be misconfigured?

> My next steps are to add simple request logging during the Trans phase,
> and failing that, packet sniffing, but this is a live high-traffic
> server so I'm trying to avoid that if possible.

Packet sniffing will answer the question "was there any evidence of a
request", and would probably be the least invasive way of measuring
this. It's always useful to see what the network is actually doing,
rather than what you think it's doing.

Regards,
Graham
--

(apologies if this is a dupe, I originally sent from the wrong address)

Hi there,

I have an unusual problem - a large e-commerce site integrated with
Authorize.net for card payments which appears to be failing to log some
requests.

The Authorize.net system makes HTTP POST requests to our server, and
about 1 in every 500 transactions, the Authorize.net system reports a
timeout and there's no trace of the request in our logs.  Authorize.net
won't investigate in any detail because their system is reporting that
the request simply timed out.

I'm using mod_perl but not hooking into the logging phase, and using a
mod_log CustomLog directive which outputs the usual stuff, plus request
time, connection state, and PID.

So for now, I'm assuming a request is being sent but the log handler
phase isn't running.  The only way I can make this happen in a test
environment is by opening a TCP connection and then closing it without
sending any data.  Are there any other reasons the log phase wouldn't be
run?

My next steps are to add simple request logging during the Trans phase,
and failing that, packet sniffing, but this is a live high-traffic
server so I'm trying to avoid that if possible.

cheers
John

On 12/02/2009 09:53 PM, Rainer Jung wrote:

> 
> Yes, that fixes it, makes a lot of sense.
> 
> What's interesting is, that the function address changes during the
> first restart, but not any more when doing further restarts. So the
> initial start ends up with a different load order than all of the
> following restarts. Something to keep in mind at least in case it's new
> in 2.3 - and about 60 other places to check which use optional
> functions, whether the function pointers are correctly reinitialized.
> 
> Thanks!
> 
> Will you commit?

Done as r886324.

Regards

Rüdiger

Hi Rüdiger,

On 02.12.2009 21:33, Ruediger Pluem wrote:
>
>
> On 12/02/2009 09:09 PM, Rainer Jung wrote:
>> Platform: Solaris 8 (sic!)
>> MPM: worker dynamically loaded
>> APR etc: Bundled
>> PCRE: 7.8
>>
>> During testing of 2.3.4 I noticed crashes after restart.
>>
>> I did a build with lots of modules, especially including mod_logio. The
>> scoreboard uses in ap_increment_counts() the optional function
>> ap_logio_get_last_bytes() from mod_logio if available.
>>
>> In my case after restart the memory location of mod_logio and the
>> address of the optional function changed, but the scoreboard still tries
>> to call to the original address retrieved after the initial start.
>>
>> I don't know about the full implementation of the optional functions,
>> but it seems either
>>
>> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
>>
>> in register_hooks in mod_logio needs to run after restarts too, or there
>> is a problem resulting in an unwanted change of load order of the
>> modules during restart. I did not edit the config files between start
>> and restart.
>>
>> The problem happens with normal restarts and graceful restarts.
>>
>> Wild guess: it might have to do with dynamic MPM loading.
>
> I am not so sure. I guess the problem is that we call
>
> APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
>
> only in ap_calc_scoreboard_size which is IMHO not called at restarts.
> Does the following totally untested patch fix your issue?
>
> Index: scoreboard.c
> ===================================================================
> --- scoreboard.c        (Revision 886294)
> +++ scoreboard.c        (Arbeitskopie)
> @@ -284,6 +284,8 @@
>       apr_status_t rv;
>   #endif
>
> +    pfn_ap_logio_get_last_bytes = APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
> +
>       if (ap_scoreboard_image) {
>           running_gen = ap_scoreboard_image->global->running_generation;
>           ap_scoreboard_image->global->restart_time = apr_time_now();
>
>

Yes, that fixes it, makes a lot of sense.

What's interesting is, that the function address changes during the 
first restart, but not any more when doing further restarts. So the 
initial start ends up with a different load order than all of the 
following restarts. Something to keep in mind at least in case it's new 
in 2.3 - and about 60 other places to check which use optional 
functions, whether the function pointers are correctly reinitialized.

Thanks!

Will you commit?

Rainer

On Wed, Dec 2, 2009 at 3:33 PM, Ruediger Pluem <rpluem@apache.org> wrote:
>
>
> On 12/02/2009 09:09 PM, Rainer Jung wrote:
>> Platform: Solaris 8 (sic!)
>> MPM: worker dynamically loaded
>> APR etc: Bundled
>> PCRE: 7.8
>>
>> During testing of 2.3.4 I noticed crashes after restart.
>>
>> I did a build with lots of modules, especially including mod_logio. The
>> scoreboard uses in ap_increment_counts() the optional function
>> ap_logio_get_last_bytes() from mod_logio if available.
>>
>> In my case after restart the memory location of mod_logio and the
>> address of the optional function changed, but the scoreboard still tries
>> to call to the original address retrieved after the initial start.
>>
>> I don't know about the full implementation of the optional functions,
>> but it seems either
>>
>> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
>>
>> in register_hooks in mod_logio needs to run after restarts too, or there
>> is a problem resulting in an unwanted change of load order of the
>> modules during restart. I did not edit the config files between start
>> and restart.
>>
>> The problem happens with normal restarts and graceful restarts.
>>
>> Wild guess: it might have to do with dynamic MPM loading.
>
> I am not so sure. I guess the problem is that we call
>
> APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
>
> only in ap_calc_scoreboard_size which is IMHO not called at restarts.

That was my guess.  Which reminds me of this old challenge for a module I wrote:

 * XXX
 * our daemon must be created in the pre-mpm hook so that the scoreboard
 * is available in the daemon process; the scoreboard isn't created until
 * the core module's pre-mpm hook
 * BUT: the pre-mpm hook is not called for graceful restart, so we can't
 *      start a new instance of the daemon then
 * BUT: request processing from the daemon MUST use the new configuration,
 *      so we'd be busted for graceful restart

On 25.11.2009 23:43, Paul Querna wrote:
> Test tarballs for Apache httpd 2.3.4-alpha are available at:
>     <http://httpd.apache.org/dev/dist/>
>
> Your votes please;
>
>   +/- 1
>   [  ]  Release httpd-2.3.4 as Alpha
>
> Vote closes at 18:00 UTC on Monday November 30 2009.
>
> May your Thanksgiving be filled with Turkey and httpd testing,

I'm +0 as an alpha.

I reported a crash a couple of minutes ago, but it's not clear at the 
moment, whether it's only a problem in mod_logio (which is rarely used) 
or a more general problem concerning optional function hooks and restarts.

Other small things:

1) Use of AP_LDAP_OPT_DEBUG in modules/ldap/util_ldap.c, already fixed 
by Günter.

2) HTTP KeepAlive: there is still a problem with FD handling (no 
surprise, because nothing changed since 2.3.1): see

   http://marc.info/?t=123102466200005&r=1&w=2

In apr configure, there are (at least) two places with

   $RM conftest*

but RM=rm and conftest* doesn't exist at that moment, so it obviously 
produces

   conftest*: No such file or directory

Line numbers are 13902 (and 13957) and also 15018. Don't know whether RM 
should be "rm -f" or instead it should "$RM -f contest*".

3) configure in the expat directory throws a syntax error:

.../srclib/apr-util/xml/expat/configure[2462]: syntax error at line 3321 
: `(' unexpected

It has to do with the function lt_if_append_uniq used there. If I 
rebuild configure using the buildconf.sh script inside the expat 
directory, the function is no longer used in the configure script and 
the error is gone. It seems the function lt_if_append_uniq() comes from 
libtool 2.2. I use 1.5.26. Looks like we have to agree on minimum 
toolchain needed and fix documentation if we upgrade to libtool 2.2 as a 
requirement.

Regards,

Rainer

On 12/02/2009 09:09 PM, Rainer Jung wrote:
> Platform: Solaris 8 (sic!)
> MPM: worker dynamically loaded
> APR etc: Bundled
> PCRE: 7.8
> 
> During testing of 2.3.4 I noticed crashes after restart.
> 
> I did a build with lots of modules, especially including mod_logio. The
> scoreboard uses in ap_increment_counts() the optional function
> ap_logio_get_last_bytes() from mod_logio if available.
> 
> In my case after restart the memory location of mod_logio and the
> address of the optional function changed, but the scoreboard still tries
> to call to the original address retrieved after the initial start.
> 
> I don't know about the full implementation of the optional functions,
> but it seems either
> 
> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
> 
> in register_hooks in mod_logio needs to run after restarts too, or there
> is a problem resulting in an unwanted change of load order of the
> modules during restart. I did not edit the config files between start
> and restart.
> 
> The problem happens with normal restarts and graceful restarts.
> 
> Wild guess: it might have to do with dynamic MPM loading.

I am not so sure. I guess the problem is that we call

APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);

only in ap_calc_scoreboard_size which is IMHO not called at restarts.
Does the following totally untested patch fix your issue?

Index: scoreboard.c
===================================================================
--- scoreboard.c        (Revision 886294)
+++ scoreboard.c        (Arbeitskopie)
@@ -284,6 +284,8 @@
     apr_status_t rv;
 #endif

+    pfn_ap_logio_get_last_bytes = APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
+
     if (ap_scoreboard_image) {
         running_gen = ap_scoreboard_image->global->running_generation;
         ap_scoreboard_image->global->restart_time = apr_time_now();


Regards

Rüdiger