lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ronnie Kolehmainen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-644) Contrib: another highlighter approach
Date Wed, 02 Aug 2006 13:55:14 GMT
    [ http://issues.apache.org/jira/browse/LUCENE-644?page=comments#action_12425237 ] 
            
Ronnie Kolehmainen commented on LUCENE-644:
-------------------------------------------

Many thanks for your feedback Mark.

I have to admit, this was a a rather quick fix for our needs, thus a separate class. Ideally,
Highlighter and TokenSources should handle this internally and transparent, when applicable.
Maybe some sort of HighlightedTokenStream (just out of my head) with query term tokens plus
a few surrounding tokens. Should be doable

I'm heading for Seattle shortly so i'm not sure I will have so much time either, but if I
do I will see if the current package can be patched in order to benefit from the speed improvements
available. Meanwhile this code can stay here as a reminder/inspiration source, or if someone
finds use for it ;)

Cheers!
Ronnie



> Contrib: another highlighter approach
> -------------------------------------
>
>                 Key: LUCENE-644
>                 URL: http://issues.apache.org/jira/browse/LUCENE-644
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Other
>            Reporter: Ronnie Kolehmainen
>            Priority: Minor
>         Attachments: FulltextHighlighter.java, FulltextHighlighterTest.java, svn-diff.patch
>
>
> Mark Harwoods highlighter package is a great contribution to Lucene, I've used it a lot!
However, when you have *large* documents (fields), highlighting can be quite time consuming
if you increase the number of bytes to analyze with setMaxDocBytesToAnalyze(int). The default
value of 50k is often too low for indexed PDFs etcetera, which results in empty highlight
strings.
> This is an alternative approach using term position vectors only to build fragment info
objects. Then a StringReader can read the relevant fragments and skip() between them. This
is a lot faster. Also, this method uses the *entire* field for finding the best fragments
so you're always guaranteed to get a highlight snippet.
> Because this method only works with fields which have term positions stored one can check
if this method works for a particular field using following code (taken from TokenSources.java):
>         TermFreqVector tfv = (TermFreqVector) reader.getTermFreqVector(docId, field);
>         if (tfv != null && tfv instanceof TermPositionVector)
>         {
>           // use FulltextHighlighter
>         }
>         else
>         {
>           // use standard Highlighter
>         }
> Someone else might find this useful so I'm posting the code here.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message