lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Commented: (LUCENE-2167) Implement StandardTokenizer with the UAX#29 Standard
Date Fri, 11 Jun 2010 16:29:13 GMT


Robert Muir commented on LUCENE-2167:

bq. NewStandardTokenizer is not quite finished; I plan on stealing Robert's Southeast Asian
(Lao, Myanmar, Khmer) syllabification routine

Curious, what is your plan here? Do you plan to somehow "jflex-#include" these into the grammar
so that these are longest-matched instead of the Complex_Context rule? 

How to handle the cases where the grammar cannot be forward-only deterministic matching? (at
least i don't see how it could be, but maybe). e.g. the lao cases where some backtracking
is needed... and the combining class reordering needed for real-world text?

Curious what would you plan to index for Thai, words? a grammar for TCC?

Also, some of these syllable techniques are probably not very good for search without doing
a "shingle" later... in some cases it may perform OK like single ideographs or tibetan syllables
do with the grammar you have. For others (Khmer, etc) I think the shingling is likely mandatory
since they are really only a bit better than indexing grapheme clusters.

As far as needing punctuation for shingling, the similar problem already exists. For example,
after tokenizing, some discarding of information (punctuation) has been lost and its too late
to do a nice shingle. practical cheating/workarounds exist for CJK (you could look at the
offset or something and cheat, to figure out that they were adjacent), but for something like
Tibetan the type of punctuation itself is important: the tsheg being unambiguous syllable
separator, but ambiguous word separator, but the shad or whitespace being both. 

Here is the paper I brought up at ehatcher's house recently when we were discussing tibetan,
that recommends this syllable bigram technique, where the shingling is dependent on the original

One alternative for the short term would be to make a tokenfilter that hooks into the ICUTokenizer
logic but looks for Complex_Context, or similar. I definitely agree it would be best if standardtokenizer
worked the best out of box without doing something like this.

Finally, I think its worth considering a lot of this as a special case of a larger problem
that affects even english. For a lot of users, punctuation such as the hyphen in english might
have some special meaning and they might want to shingle or something else in that case too.
Its a general problem with tokenstreams that the tokenizer often discards this information
and the filters are left with only a partial picture. Some ideas to improve it would be to
make use of properties like [:Terminal_Punctuation=Yes:] somehow, or to try to integrate Sentence

> Implement StandardTokenizer with the UAX#29 Standard
> ----------------------------------------------------
>                 Key: LUCENE-2167
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: contrib/analyzers
>    Affects Versions: 3.1
>            Reporter: Shyamal Prasad
>            Assignee: Steven Rowe
>            Priority: Minor
>         Attachments: LUCENE-2167-jflex-tld-macro-gen.patch, LUCENE-2167-jflex-tld-macro-gen.patch,
LUCENE-2167-jflex-tld-macro-gen.patch, LUCENE-2167-lucene-buildhelper-maven-plugin.patch,
LUCENE-2167.benchmark.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch,
LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch,
LUCENE-2167.patch, LUCENE-2167.patch, LUCENE-2167.patch
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
> It would be really nice for StandardTokenizer to adhere straight to the standard as much
as we can with jflex. Then its name would actually make sense.
> Such a transition would involve renaming the old StandardTokenizer to EuropeanTokenizer,
as its javadoc claims:
> bq. This should be a good tokenizer for most European-language documents
> The new StandardTokenizer could then say
> bq. This should be a good tokenizer for most languages.
> All the english/euro-centric stuff like the acronym/company/apostrophe stuff can stay
with that EuropeanTokenizer, and it could be used by the european analyzers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message