lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-966) A faster JFlex-based replacement for StandardAnalyzer
Date Wed, 01 Aug 2007 15:50:52 GMT


Michael McCandless commented on LUCENE-966:

Oddly, the patch for TestStandardAnalyzer failed to apply for me (but
the rest did), so I manually merged those changes in.  Oh, I see: it
was the "Korean words" test -- somehow the characters got mapped to
?'s in your patch.  This is why the patch didn't apply, I think?
Maybe you used a diffing tool that wasn't happy with unicode or

I also see the quality test failing in contrib benchmark.  I fear
something about the new StandardAnalyzer is in fact causing this test
to fail (it passes on a clean checkout).  That test uses

KO I re-tested the old vs new StandardAnalyzer on Wikipedia and I
still found some differences, I think only on these very large
URL-like tokens.  Here's one:



I like the NEW behavior better but I fear we should try to match the
old one?

Here's another one:



I like the old behavior better here.

Another one:



Another one:



> A faster JFlex-based replacement for StandardAnalyzer
> -----------------------------------------------------
>                 Key: LUCENE-966
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>            Reporter: Stanislaw Osinski
>             Fix For: 2.3
>         Attachments:, jflex-analyzer-patch.txt, jflex-analyzer-r560135-patch.txt,
jflex-analyzer-r561292-patch.txt, jflex-analyzer-r561693-compatibility.txt
> JFlex ( can be used to generate a faster (up to several times) replacement
for StandardAnalyzer. Will add a patch and a simple benchmark code in a while.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message