lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2939) Highlighter should try and use maxDocCharsToAnalyze in WeightedSpanTermExtractor when adding a new field to MemoryIndex
Date Sun, 27 Feb 2011 01:43:59 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999863#comment-12999863
] 

Mark Miller commented on LUCENE-2939:
-------------------------------------

hmmm...didn't quite get it right yet I think...

q=java man news th*


{code}
java.lang.NullPointerException
	at org.apache.lucene.util.CharacterUtils$Java5CharacterUtils.fill(CharacterUtils.java:181)
	at org.apache.lucene.analysis.CharTokenizer.incrementToken(CharTokenizer.java:150)
	at org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:224)
	at org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:54)
	at org.apache.lucene.analysis.miscellaneous.ASCIIFoldingFilter.incrementToken(ASCIIFoldingFilter.java:71)
	at com.ACME.analysis.ACMEPluralStemFilter.incrementToken(ACMEPluralStemFilter.java:56)
	at org.apache.solr.highlight.TokenOrderingFilter.incrementToken(DefaultSolrHighlighter.java:575)
	at org.apache.lucene.analysis.CachingTokenFilter.fillCache(CachingTokenFilter.java:78)
	at org.apache.lucene.analysis.CachingTokenFilter.incrementToken(CachingTokenFilter.java:50)
	at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:220)

{code}

> Highlighter should try and use maxDocCharsToAnalyze in WeightedSpanTermExtractor when
adding a new field to MemoryIndex
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-2939
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2939
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: contrib/highlighter
>            Reporter: Mark Miller
>            Assignee: Mark Miller
>            Priority: Minor
>         Attachments: LUCENE-2939.patch
>
>
> huge documents can be drastically slower than need be because the entire field is added
to the memory index
> this cost can be greatly reduced in many cases if we try and respect maxDocCharsToAnalyze
> the cost is still not fantastic, but is at least improved in many situations and can
be influenced with this change

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message