apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William A Rowe Jr <wr...@rowe-clan.net>
Subject Fwd: apr_token_* conclusions (was: Better casecmpstr[n]?)
Date Wed, 25 Nov 2015 17:45:02 GMT
tl;dr - feel free to skip "httpd" specifics to get to the meat of the APR
discussion items.

---------- Forwarded message ----------
From: William A Rowe Jr <wrowe@rowe-clan.net>
Date: Wed, Nov 25, 2015 at 11:42 AM
Subject: apr_token_* conclusions (was: Better casecmpstr[n]?)
To: httpd <dev@httpd.apache.org>

On Wed, Nov 25, 2015 at 10:17 AM, Jim Jagielski <jim@jagunet.com> wrote:

> What is the current status? Is this on hold?

It is looking for a good name.  I'm happy with apr_token_strcasecmp
to best indicate its use-case and provenance.  Does that work for

It is looking for clearer docs.  Spent 20 hours just reviewing locale in C
(partly to phrase this discussion more accurately). About to pen those
based on better definitions of terms.

So here are my conclusions as they apply to apr, and to httpd after
all of this locale review...


The C spec defines the locale as "C" (e.g. "POSIX") at startup.  Until
somewhere in the code the locale of the current thread is switched with
a call to setlocale(LC_ALL, "") or similar, the application remains in this
non-deterministic (Anglicized) state.  The empty string causes LANG
and the LC_* variables to be evaluated.  Most modern *nix utilities do
this right off the bat in their source.  The compiler should *not* do so
without instruction, so while gcc does the right thing in this respect,
other compilers may not be behaving appropriately.

You might react "how do we handle UTF-8 [or other code page] then?"
The answer is that in the "C" locale, high-bit characters (in ASCII, and
even unusual EBCDIC codes) are effectively opaque.  They *can* be
UTF-8, they might belong to an SBCS (single-byte charset), and they
might be entirely meaningless.  They won't be case folded (but will
keep their unique identity).  Some consumer may recognize them,
others will not.  The C lib functions will treat each octet of an MBCS
as a distinct character, which is a reason that our old-school autoindex
(pre-fancy tables) misaligns the columns when the filename contains
any multibyte characters.  We had simply byte-counted chars.

If our code splits these multibyte sequences, things can go wrong
somewhere down the way.  In particular, treating these multibyte
sequences as distinct characters is fine in UTF-8 where every part
of the multibyte sequence is high-bit-set (and therefore opaque),
but is not fine in ISO2022-JP, where low-bit-set characters may
change their meaning (there are bugs to the effect that we search
a string for pathname separator characters, and these can occur
in multibyte sequences which are valid file names, and not path
separator characters).  There isn't much we can do about this.

When using httpd on a *nix filesystem containing UTF-8 chars, we
accept UTF-8 filenames both in client provided fields and within the
httpd.conf configuration.  On a filesystem containing SBCS filenames,
we similarly accept these without translation.  It's up to the admin
to decide their schema, without a third party module, the UTF-8
name isn't recognized in an SBCS directory, and visa-versa.

On Windows, all system resources, including filenames, are stored
in Unicode by the OS.  Within APR, we simply treat all system
resource strings as UTF-8, and therefore the conf file in httpd needs
to spell out the UTF-8 name of the resource. All that said, other than
these resource names, even on Windows most string processing
follows the same opaque logic as *nix.

The Mac OS is somewhat similar to Windows, in that all of the
resource names are actually UTF-8 encoded, AFAICT. But unlike
windows, these opaque strings "just work", we don't have to do any
Unicode transliteration to get there.


Nowhere in apr or httpd do *we* call setlocale() to change things.  So
the current use of tolower(), toupper(), strcasecmp() etc should not be
subject to dangerous transliterations by *our* doing.

There is nothing that suggests that the APR consumer *cannot*
call setlocale()!  So we need to revisit APR 2.0 and ensure that
our functions perform as-expected even when operating under
some different locale.  This suggests that apr_token_str[n]casecmp,
apr_token_tolower|toupper, and much of the rest of the code that
exists only to evaluate ASCII alpha characters is *not* paying any
attention to the currently defined locale.

In httpd, where things go sideways is if someone is calling setlocale(),
for example in an in-process PHP, Perl or Lua script, because this
changes the core operation of httpd.  If the script switches setlocale
to turkish, for example, our forced-lowercase content-type conversion
will cause "IMAGE/GIF" to become "ımage/gıf", clearly not what the
specs intended.

I imagine that this has rarely shown up because the few scripts that
might toggle this were correctly written to toggle back the previous
locale upon completion.  But if they are running inline and have
the locale toggled during the filter handoff, the filters themselves
are likely facing some unexpected behavior.

APR conclusions

Adding unambiguous token handling functions would be good for
the few case-insensitive string comparison, string folding, and
search functions.  It allows the spec-consumer to trust their string
processing.  I'm going to suggest apr_token_* as the API prefix.
The API will preserve "POSIX behavior" as defined by A-Z <> a-z
character equivalence and make no compensation for locales
or for MBCS strings.

httpd conclusions

There are so many edge cases that we simply need to preserve
the code as-is in httpd 2.4 and warn off anyone toggling the locale
that httpd is operating under.  Making the 'offer' to run under many
non-POSIX locales is opening up a can of security vulnerabilities.
It would be irresponsible to half-fix this.

We should perform a thorough review of httpd 2.x trunk so as to
make a statement upon release that the code has been adapted
to correctly operate under most non-POSIX locales.  Warn off
users from third party modules that have not yet made a similar
statement.  Kill all the redundant httpd 2.x == APR functions that
do not belong in trunk (but may have been appropriate in 2.earlier
while waiting for APR to be released).

To the extent that implementations have very poor implementations
of strcmp(), this isn't justification to overload httpd with more band-aids.
Fix the underlying implementation.  So I'm -0.5 on backporting this
change into httpd 2.4 until we see a comprehensive justification, or
until we comprehensively have fixed trunk and are prepared to make
the same "setlocale()-safe" assertion about 2.4.future.

View raw message