lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rowe (JIRA)" <>
Subject [jira] Commented: (LUCENE-2167) Implement StandardTokenizer with the UAX#29 Standard
Date Thu, 27 May 2010 13:54:28 GMT


Steven Rowe commented on LUCENE-2167:

bq. Must it be a maven plugin? [...] Its much easier from ant to invoke a java method as a
build step.

Lucene's build could be converted to Maven, though, and this could be a place for build-related

Maven Ant Tasks allows for Ant to call full Maven builds without a Maven installation:

bq. From what I see, the same code could be done as a simple Java Class with main() like Roberts
ICU converter. [snip]

I hadn't seen Robert's ICU converter - I'll take a look.

bq. A python script may also work well.

Perl is my scripting language of choice, not Python, but yes, a script would likely do the
trick, assuming there are no external (Java) dependencies.  (And as you pointed out, HttpComponents,
the only dependency of the Maven plugin, does not need to be a dependency.)

bq. You use the HTML root zone database from IANA. The format of this file is hard to parse
and may change suddenly. BIND administrators know, that there is also the root zone file available
for BIND in the standardized named-format @ (ASCII
only, as DNS is ASCII only).

I think I'll stick with the HTML version for now - there are no decoded versions of the internationalized
TLDs and no descriptive information in the named-format version.  I agree the HTML format
is not ideal, but it took me just a little while to put together the regexes to parse it;
when the format changes, the effort to fix will likely be similarly small.

bq. You can write the Last-Modified-Header of the HTTP-date (HttpURLConnection.getLastModified())
also into the generated file.

Excellent idea, I searched the HTML page source for this kind of information but it wasn't

bq. The database only contains the punycode enabled DNS names. But users use the non-encoded
variants, so you should decode punycode, too [we need ICU for that :( ] and create patterns
for that, too.

I agree.  However, I looked into what's required to do internationalized domain names properly,
and it's quite complicated.  I plan on doing what you suggest eventually, both for TLDs and
all other domain labels, but I'd rather finish the ASCII implementation and deal with IRIs
in a separate follow-on issue.

bq. About changes in analyzer syntax because of regeneration: This should not be a problem,
as the IANA only adds new zones to the file and very seldom removes some (like old yugoslavian
zones). As eMails and Webadresses should not appear in tokenized text before they are in the
zone file, its no problem that they suddenly later are marked as "URL/eMail" (as they cannot
appear before). So in my opinion we can update the zone database even in minor Lucene releases
without breaking analyzers.


> Implement StandardTokenizer with the UAX#29 Standard
> ----------------------------------------------------
>                 Key: LUCENE-2167
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: contrib/analyzers
>    Affects Versions: 3.1
>            Reporter: Shyamal Prasad
>            Assignee: Steven Rowe
>            Priority: Minor
>         Attachments: LUCENE-2167-lucene-buildhelper-maven-plugin.patch, LUCENE-2167.benchmark.patch,
LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch,
LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
> It would be really nice for StandardTokenizer to adhere straight to the standard as much
as we can with jflex. Then its name would actually make sense.
> Such a transition would involve renaming the old StandardTokenizer to EuropeanTokenizer,
as its javadoc claims:
> bq. This should be a good tokenizer for most European-language documents
> The new StandardTokenizer could then say
> bq. This should be a good tokenizer for most languages.
> All the english/euro-centric stuff like the acronym/company/apostrophe stuff can stay
with that EuropeanTokenizer, and it could be used by the european analyzers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message