lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <>
Subject [jira] Commented: (LUCENE-966) A faster JFlex-based replacement for StandardAnalyzer
Date Thu, 02 Aug 2007 20:43:53 GMT


Mark Miller commented on LUCENE-966:

>When digging deeper into the issues of compatibility with the original StandardAnalyzer,
I stumbled upon something strange. Take the following >text:
>which was tokenized by the original StandardAnalyzer as one <NUM>. If you look at
the definition of the <NUM> token:
>// every other segment must have at least one digit
>       | <HAS_DIGIT> <P> <ALPHANUM>
>       | <ALPHANUM> (<P> <HAS_DIGIT> <P> <ALPHANUM>)+
>       | <HAS_DIGIT> (<P> <ALPHANUM> <P> <HAS_DIGIT>)+
>       | <ALPHANUM> <P> <HAS_DIGIT> (<P> <ALPHANUM> <P>
>       | <HAS_DIGIT> <P> <ALPHANUM> (<P> <HAS_DIGIT> <P>
>        )
>you'll see that, as explained in the comment, every other segment must have at least one
digit. But actually, according to my understanding, this >rule should not match the above
text as a whole (and with JFlex it doesn't , actually). Below is the text split by punctuation
characters, and it looks >like there is no way of splitting this text into alternating
segments, every second of which must have a digit (A = ALPHANUM, H = HAS_DIGIT):
>78academyawards / rules / rule02 . html , 7194 , 7227 , type
>                H P A P H P A P H P A P H?* (starting from the beginning)
>                                                                              H?* P A
P H P A (starting from the end)
>* (would have to be H, but no digits in substring "type" or "html")
>I have no idea why JavaCC matched the whole text as a <NUM>, JFlex behaved "more
correctly" here. 

I think that JavaCC is correct here. Every other segment must have a digit: 78academyawards
/ rules / rule02 . html 
It looks to me like every other segment does have a digit. First segment has a digit, second
does not, third does, fourth does not. Matches NUM correctly.

> A faster JFlex-based replacement for StandardAnalyzer
> -----------------------------------------------------
>                 Key: LUCENE-966
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>            Reporter: Stanislaw Osinski
>             Fix For: 2.3
>         Attachments:, jflex-analyzer-patch.txt, jflex-analyzer-r560135-patch.txt,
jflex-analyzer-r561292-patch.txt, jflex-analyzer-r561693-compatibility.txt
> JFlex ( can be used to generate a faster (up to several times) replacement
for StandardAnalyzer. Will add a patch and a simple benchmark code in a while.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message