lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Commented: (LUCENE-2265) improve automaton performance by running on byte[]
Date Fri, 02 Apr 2010 18:06:27 GMT


Robert Muir commented on LUCENE-2265:

I discussed this situation with Mike McCandless and I think we might have something of a plan.

For reference, here is the problem: 
* In flex the terms are byte[] (typically UTF-8)
* Automaton transitions work on UTF-16 intervals (char)
* RunAutomaton is an array-compiled form that also works on UTF-16 (char[])
* Because of this, we have a lot of unicode conversion overhead between byte[] and char[]
hurting performance.

Here is the current idea:
* Switch Automaton to work on UTF-32 intervals (int)
* Create a method to convert a UTF-32 Automaton to an equivalent UTF-8 Automaton.
* Create a UTF-8 RunAutomaton that works on byte[]
* We could also create a UTF-32 RunAutomaton that works on codepoints, for use in analysis,

This would have some nice benefits besides performance, 
for example a wildcard operator of "?" or regex operator of "." would match a real unicode
not a single code unit like it always has. So if somehow we can make this work, we would have
performance and better unicode support.

The trick is to do this UTF-32 DFA -> UTF-8 DFA conversion efficiently, especially keeping
and not causing some nasty explosion

> improve automaton performance by running on byte[]
> --------------------------------------------------
>                 Key: LUCENE-2265
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>    Affects Versions: Flex Branch
>            Reporter: Robert Muir
>            Priority: Minor
>             Fix For: Flex Branch
>         Attachments: LUCENE-2265.patch
> Currently, when enumerating terms, automaton must convert entire terms from flex's native
utf-8 byte[] to char[] first, then step each char thru the state machine.
> we can make this more efficient, by allowing the state machine to run on byte[], so it
can return true/false faster.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message