lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-966) A faster JFlex-based replacement for StandardAnalyzer
Date Thu, 02 Aug 2007 19:32:52 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517348
] 

Mark Miller commented on LUCENE-966:
------------------------------------

These issues seem odd.

Both JavaCC and Flex match with the same rules:
1. Longest match first
2. If match size is the same, use the first in the grammar

OLD
    (money.cnn.com,1382,1395,type=<HOST>)
    (magazines,1396,1405,type=<ALPHANUM>)
    (fortune,1406,1413,type=<ALPHANUM>)
    (fortune,1414,1421,type=<ALPHANUM>)
    (archive/2007/03/19/8402357,1422,1448,type=<NUM>)
    (index.htm,1449,1458,type=<HOST>)

  NEW
    (/money.cnn.com/magazines/fortune/fortune_archive/2007/03/19/8402357/index.htm,1381,1458,type=<NUM>)

The old is correct and the NEW should not match <NUM>. <NUM> should break on '/'
and '.' and every other token from the break should have a digit for a NUM match to occur.
This is not the case.

  OLD
    (mid-20th,2436,2444,type=<NUM>)

  NEW
    (mid,2436,2439,type=<ALPHANUM>)
    (-20th,2439,2444,type=<NUM>)

Something is wrong with the NEW one. <NUM> is certainly a valid longer match.

  OLD
    (safari-0-sheikh,12011,12026,type=<NUM>)
    (zayed,12027,12032,type=<ALPHANUM>)
    (grand,12033,12038,type=<ALPHANUM>)
    (mosque.jpg,12039,12049,type=<HOST>)

  NEW
    (safari,12011,12017,type=<ALPHANUM>)
    (0-sheikh-zayed-grand-mosque.jpg,12018,12049,type=<NUM>)

Again, something seems wrong with the NEW.  (safari-0-sheikh,12011,12026,type=<NUM>)
is a correct and longer match than  (safari,12011,12017,type=<ALPHANUM>)

It would be nice to have the source text for these comparisons.

Also, a hard vote against StandardAnalyzer2 <g> Default is arguable as well, as this
wouldn't be the default analyzer you should use in many cases (don't like standard because
of that either).

>From the latest samples, I would say something is off with the NEW and OLD appears mostly
correct.

- Mark


> A faster JFlex-based replacement for StandardAnalyzer
> -----------------------------------------------------
>
>                 Key: LUCENE-966
>                 URL: https://issues.apache.org/jira/browse/LUCENE-966
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>            Reporter: Stanislaw Osinski
>             Fix For: 2.3
>
>         Attachments: AnalyzerBenchmark.java, jflex-analyzer-patch.txt, jflex-analyzer-r560135-patch.txt,
jflex-analyzer-r561292-patch.txt, jflex-analyzer-r561693-compatibility.txt
>
>
> JFlex (http://www.jflex.de/) can be used to generate a faster (up to several times) replacement
for StandardAnalyzer. Will add a patch and a simple benchmark code in a while.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message