stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Sebor <>
Subject Re: [Stdcxx Wiki] Update of "LocaleLookup" by MartinSebor
Date Thu, 13 Mar 2008 04:34:42 GMT
Travis Vitek wrote:
>> I think it might be simpler to keep things abstract but given my
>> specification above a simple query string would look like this:
>>    "en_US.*    1\n"
>>    "zh_*.UTF-8 2\n"
>>    "zh_*.UTF-8 3\n"
>>    "zh_*.UTF-8 4\n"
>> for the equivalent of:
>>       locale == "en_US.*"    && MB_CUR_MAX == 1
>>    || locale == "zh_*.UTF-8" && MB_CUR_MAX == 2
>>    || locale == "zh_*.UTF-8" && MB_CUR_MAX == 3
>>    || locale == "zh_*.UTF-8" && MB_CUR_MAX == 4
> I'm totally confused. If we're going to write out each of the expansions,
> then why did I take the time to implement brace expansion?
> I thought the idea was to allow us to select locales using a brace expanded
> query string. If we are explicitly writing out each of the names, then we
> wasted a bunch of time writing brace expansion code.

Yes, the idea was/is to use brace expansion to provide alternation.
I'm not dismissing the idea, just observing that it doesn't neatly
fit into the grammar I outlined above in the absence of anything
else. Can you show your grammar and explain how it works so that
I can better understand what we're discussing?

>> I'm not sure how you could use brace expressions here. Maybe it
>> should be the other way around (<SP> should be OR and <NL> AND).
>> But then the grep -e idea is out the window.
> Well, if we're going down the road of rewriting this _again_

Usually design precedes implementation. If you opt to implement first
you need to be prepared to make changes based on flaws in your (or
anyone else's) ideas that you use to ground your implementation in.

> then how
> about using something like '&&' and '||', or even 'and' and 'or' for the
> logical operations and then '(' and ')' for grouping? Almost like the
> 'equivalent of' that you wrote above. Something that is readable by a
> C/C++ programmer or the average guy off of the street?

We discussed and rejected this syntax.

> The truth is that not every guy knows grep,

Anyone who works with this aspect of stdcxx will need to know it.

> and I'm sure that those who
> do wouldn't expect to see a grammar that used '\n' and ' ' to represent
> logical operations.

I agree. I will welcome a better alternative that's in line with
the "grep" spirit of the solution we agreed on (so that it can
be made use of in the Expected Failures project).

>> Or maybe we need
>> a way to denote/group terms. Then we might be able to say:
>>    "en_US.*    1\n"
>>    "zh_*.UTF-8 ({2..4})"
>> expand it to
>>    "en_US.*    1\n"
>>    "zh_*.UTF-8 (2 3 4)"
>> and "know" that the spaces in "2 3 4" denote OR even though the
>> space in "zh_*.UTF-8 (" denotes an AND. Ugh. I'm not sure I lik
>> this all that much better.
>>>   const char* locales =
>>>       rw_locale_query ("en_US.* 1\nzh_*.UTF-8 {2..4}", 10);
>>> I don't see why that would be necessary. You can do it with the
>>> following query using normal brace expansion, and it's human readable.
>>>   const char* locales =
>>>       rw_locale_query ("{en_US.* 1,zh_*.UTF-8 {2..4}}", 10);
>> What's "{en_US.* 1,zh_*.UTF-8 {2..4}}" supposed to expand to?
>> Bash 3.2 doesn't expand it. I suppose it could be
>>    "en_US.* 1 zh_*.UTF-8 2 3 4" or
>> "en_US.* 1 zh_*.UTF-8 2 zh_*.UTF-8 3 zh_*.UTF-8 4"
> I believe that this is _exactly_ what you suggested in our meetings when
> I was in Boulder the last time. Maybe I'm just confused, but I am pretty
> sure that was what was presented.

The whiteboard has been wiped clean so unless you wrote it down
we'll never know because I myself don't remember anymore. But if
you re-read the "long hanging fruit" thread I also suggested that
for example this string:

     "*_{JP,CN}.* {3,4}"

expand into

     "*_JP.* 4\n*_CN.* 4\n*_JP.* 3\n*_CN.* 3\n"

and probably a whole number of other things before then. The point
is that it was just a suggestion and not a fully baked design, and
like many other initial ideas it might have been flawed in more
than one way. Which is why we need to come up with a design spec
first, review it, and implement only if it makes sense to everyone
and if it deals with the use cases we're interested in dealing with.

> The shell does whitespace collapse and tokenization before it does the
> expansion. To use whitespace in a brace expansion in the shell you have
> to escape it. 
> So the following expansion should work just fine in csh...
>   {en_US.*\ 1,zh_*.UTF-8\ {2..4}}
> It should expand to
>   en_US.* 1
>   zh_*.UTF-8 2
>   zh_*.UTF-8 3
>   zh_*.UTF-8 4

With newlines at the ends or without? I assume without, and that's
precisely why I'm not happy with it. Because the string above really
     "en_US.* 1 zh_*.UTF-8 2 zh_*.UTF-8 3 zh_*.UTF-8 4"

where each odd space means AND and every even space means OR. That
seems very unintuitive to me.

> Remember that I originally provided rw_brace_expand() that doesn't do all
> of that. It treats whitespace like any other character. Of course if you
> insist on 100% compatibility with shell brace expansion, then feel free to
> escape the spaces. Personally I prefer strings without escapes.

So the explicit spaces aren't special but the ones that result from
brace expansion are? (This is an honest question.)

>> Either way I think I'm getting confused by the lack of distinction
>> between what's OR and what's AND.
> I give an example above of how a brace expansion already solves the
> problem.
> If the brace expansion routine I've written returns a null terminated
> buffer of null terminated strings that are the brace expansions and we
> have a function for doing primitive string matching [rw_fnmatch], then
> this is a pretty simple problem to solve.
> This is exactly what you are doing with the xfail.txt thing. The platform
> string is just a brace expansion and grep-like expression...
>   aix-5.3-*-vacpp-9.0-{12,15}?
> Why can't ours be seperated by spaces, or some other character? Is it so
> different?

You mean spaces instead of the dashes above? Because brace expansion
also produces spaces and because brace expansion happens before any
other processing. How would the "other processing" distinguish our
spaces from those produced from brace expansion?

Maybe we should ask the question the other way: why can't we use
dashes (or some other non-delimiting characters) in the locale spec
instead of spaces? Would that make me remove my objection? (I think
it might but I'd want to think about it some more).

> I suppose the big difference is that the format above is rigid and well
> defined, whereas the locale match format is still in flux.
>>> I know that the '\n' is how you'd use `grep -e', but does it really make
>>> sense? We aren't using `grep -e' here.
>> I'm trying to model the interface on something we all know how
>> to use. grep -e seemed the closest example of an known interface
>> that would let us do what we want that I could think of.
>> Maybe it would help to go back to the basics and try to approach
>> this by analyzing the problem first. Would putting this together
>> be helpful?
> That depends on how you define helpful. It will not be helpful in getting
> this task done in reasonable time. 

That depends on how you define reasonable ;-) I had hoped we'd be
done in two to four weeks when we started.

> It may be helpful in convincing me to
> reimplement this functionality for a third time.
>>    1. the set of locale attributes we want to keep track of
>>       in our locale "database"
> What details are necessary to reduce the number of locales tested? The
> honest answer to this is _none_. We could just pick N random locales and
> run the test with them. That would satisfy the original issue of testing
> to many locales.

But it would, in some cases, most likely compromise the effectiveness
of the tests, and if truly random, make failures hard to reproduce.

> That idea has been discarded,  so the next best thing to do is to have it
> include a list of installed locales, and the language, territory and
> codeset canonical names as well as the MB_CUR_LEN value for each. Those
> are the only criteria that we currently use for selecting locales in the
> tests.

Sounds reasonable. I see you added to the wiki. Great! What about
the OS each native name goes with? (e.g., utf8 on HP-UX vs UTF-8
on Linux). And locale aliases (e.g., en -> english -> en_US ->
en_US.ISO8859-1 on Linux -- I made up the names but the concept
is real)?

> I don't see anything else useful. If there is some detail that is useful,
> most likely we could check it by loading the locale and getting the data
> directly instead of caching that data ourselves.
>>    2. one or more possible formats of the database
> Because of all of the differences between similarly named locales on
> different systems, I don't think it makes sense to keep the locale
> data in revision control. It should probably be generated at runtime
> and flushed to a file for reuse by later tests.

Maybe. That to me seems like an implementation detail, but...

> Given that, I don't feel that the format of the data is significant. It
> might be nice for it to be human readable, but that is about it.

...this seems relevant because it will dictate the structure of
the query. Unless you want to do a whole lot of massaging of the
query before you apply it to the file. And if you take the
prototype suggestion I made -- to implement it first using grep
or some such standard tool -- it will matter a whole lot. Also
if you want to get optimum performance out of your code and
minimize its complexity the format will matter. Structure it
too much and you'll end up writing a database API to parse it.

>>    3. the kinds of queries done in our locale tests, and the
>>       ones we expect to do in future tests
> This is the important question. As mentioned above, the only thing that
> I see being used is selecting locales by name by MB_CUR_LEN.

The codecvt or ctype tests do that. The numeric, monetary, and
time tests will probably be interested in the language (e.g.,
find any German locale so that I use German days of months to
test time_put). I'm saying we should try to come up with the
*actual* queries for these tests to see if what we're doing
here will be helpful there.

>> With that, we can create a prototype solution using an existing
>> query language of our choice (such as grep). Once that works,
>> the grammar should naturally fall out and we can reimplement
>> the prototype in the test driver.
> Isn't that what you did while I was in Boulder? That is how we arrived
> at this system of brace expansion and name matching that we are talking
> about now.

By prototype I mean a solution that actually works. It could be
written in shell or Python and work on just a single platform,
and with limitations, but it proves the concept. What we did
when you visited in Boulder was sketch out a rough idea on the

> Your prototype boils down to something like this, where the fictional
> 'my_locale' utility lists the names of all installed locales followed
> by a seperator and then the MB_CUR_LEN value.
>   for i in `echo $brace_expr`;
>   do
>     my_locale -a | grep -e $i
>   done

This is a sketch, not a prototype. I can't take it and run it to
see how it works. There is no locale file so I have no idea what
$i should look like.

> Honestly, I don't care what the grammar is. I don't care what the format
> of the file is, and I don't care what shell utility we are trying to fake
> today.
> All I care about is finishing up this task. Two months is more than enough
> time for something like this to be designed and implemented.

Yet we still don't have the final grammar. It seems to me that we
need to start with it. I don't see how anyone could write any amount
of code unless they know what the grammar looks like.


View raw message