Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Sat, 2 Apr 2011 11:54:05 +0000 (UTC)
From: "Dawid Weiss (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <1719607687.30497.1301745245813.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <2069473613.7815.1298365298510.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (SOLR-2378) FST-based Lookup (suggestions) for
 prefix matches.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SOLR-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015011#comment-13015011 ] 

Dawid Weiss commented on SOLR-2378:
-----------------------------------

Soliciting feedback on the following questions:
- suggesters currently have float weights associated with terms; can these floats be bucketed and returned in approximation or do they need to be exact copies of the input? For automata, bucketed weights (to, let's say 5-10 different values) provide terrific speed/size improvements, so if this is not a rigid requirement, I'd use them.
- is there any info on threading of Solr components? I am in particular looking for mutable object fields in the suggester (can a single suggester instance be accessed by multiple threads at the same time)?

I've implemented preliminary FST-based lookup (without weights yet). Speed-wise it doesn't rock because data is converted to/from utf8 on input/output and sorted during construction, but it is still acceptable, even at this early stage I think:

{noformat}
JaspellLookup        buildTime[ms]=112 lookupTime[ms]=288
TSTLookup            buildTime[ms]=115 lookupTime[ms]=103
FSTLookup            buildTime[ms]=464 lookupTime[ms]=145
{noformat}

now... that was speed only, check out the in-memory size :)

{noformat}
JaspellLookup        size[B]=81,078,997
TSTLookup            size[B]=53,453,696
FSTLookup            size[B]=2,909,396
{noformat}

(This benchmark stores very limited vocabulary items -- long numbers only, so it is skewed from reality, but it's nice to see something like this, huh?).


> FST-based Lookup (suggestions) for prefix matches.
> --------------------------------------------------
>
>                 Key: SOLR-2378
>                 URL: https://issues.apache.org/jira/browse/SOLR-2378
>             Project: Solr
>          Issue Type: New Feature
>          Components: spellchecker
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>              Labels: lookup, prefix
>             Fix For: 4.0
>
>
> Implement a subclass of Lookup based on finite state automata/ transducers (Lucene FST package). This issue is for implementing a relatively basic prefix matcher, we will handle infixes and other types of input matches gradually. Impl. phases:
> - write a DFA based suggester effectively identical to ternary tree based solution right now,
> - baseline benchmark against tern. tree (memory consumption, rebuilding speed, indexing speed; reuse Andrzej's benchmark code)
> - modify DFA to encode term weights directly in the automaton (optimize for onlyMostPopular case)
> - benchmark again
> - add infix suggestion support with prefix matches boosted higher (?)
> - benchmark again
> - modify the tutorial on the wiki [http://wiki.apache.org/solr/Suggester]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org