lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1286) LargeDocHighlighter - another span highlighter optimized for large documents
Date Mon, 10 Nov 2008 14:12:44 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646266#action_12646266
] 

Mark Miller commented on LUCENE-1286:
-------------------------------------

It didn't turn out as well as I had hoped. You had to pay too much for the Memory index /
getting spans. I havn't closed the issue because I hope to keep trying, but I don't have anything
great at the moment. If I have the time I'll get back into it though. Storing position/offset
termvectors is the only helpful thing for large docs that I know of at the moment.

There is another highlighter by Ronnie something in JIRA that also takes this approach but
 without Phrase/Span support and requiring stored termvectors. You might try it though.

> LargeDocHighlighter - another span highlighter optimized for large documents
> ----------------------------------------------------------------------------
>
>                 Key: LUCENE-1286
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1286
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/highlighter
>    Affects Versions: 2.4
>            Reporter: Mark Miller
>            Priority: Minor
>
> The existing Highlighter API is rich and well designed, but the approach taken is not
very efficient for large documents.
> I believe that this is because the current Highlighter rebuilds the document by running
through and scoring every every token in the tokenstream.
> With a break in the current API, an alternate approach can be taken: rebuild the document
by running through the query terms by using their offsets. The benefit is clear - a large
doc will have a large tokenstream, but a query will likely be very small in comparison.
> I expect this approach to be quite a bit faster for very large documents, while still
supporting Phrase and Span queries.
> First rough patch to follow shortly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message