lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2098) make BaseCharFilter more efficient in performance
Date Tue, 16 Mar 2010 10:13:27 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845788#action_12845788
] 

Michael McCandless commented on LUCENE-2098:
--------------------------------------------

Ahh ok.

Probably we should switch to parallel arrays here, to make it very fast... yes this will consume
RAM (8 bytes per position, if we keep all of them).

Really most apps do not need all positions stored, ie, they only need to see typically the
current token.  So maybe we could make a filter that takes a "lookbehind size" and it'd only
keep that number of mappings cached?  That'd have to be > the max size of any token you
may analyze, so hard to bound perfectly, but eg setting this to the max allowed token in IndexWriter
would guarantee that we'd never have a miss?

For analyzers that buffer tokens... they'd have to set this max to infinity, or, ensure they
remap the offsets before capturing the token's state?

> make BaseCharFilter more efficient in performance
> -------------------------------------------------
>
>                 Key: LUCENE-2098
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2098
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>    Affects Versions: 3.1
>            Reporter: Koji Sekiguchi
>            Priority: Minor
>         Attachments: LUCENE-2098.patch
>
>
> Performance degradation in Solr 1.4 was reported. See:
> http://www.lucidimagination.com/search/document/43c4bdaf5c9ec98d/html_stripping_slower_in_solr_1_4
> The inefficiency has been pointed out in BaseCharFilter javadoc by Mike:
> {panel}
> NOTE: This class is not particularly efficient. For example, a new class instance is
created for every call to addOffCorrectMap(int, int), which is then appended to a private
list. 
> {panel}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message