Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.uk;
  h=Received:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding;
  b=ih7rfJjN4S2mohXzV6rr12SeOLZd3G5zKv0KNFQczD+HPEgmFPDN/pORs22mU7j2hXjXFNkuqccjt7tulNzWJ3+71V8cAo7wAsu1K0ZnzKpUwPqWkTB/RihwZ8rmXQ1ATOePbLK812+d9+gzFyDfsc+6kE2HZC8ccoVei9hKQJc=
  ;
Message-ID: <4988AEA1.2060805@yahoo.co.uk>
Date: Tue, 03 Feb 2009 20:52:49 +0000
From: markharw00d <markharw00d@yahoo.co.uk>
User-Agent: Thunderbird 2.0.0.19 (Windows/20081209)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Poor QPS with highlighting
References: <652776090902022324g7af80223pa7a4c3fe396c5b59@mail.gmail.com>
	 <7770.55606.qm@web26002.mail.ukl.yahoo.com>
 <652776090902031140h370cb16fo600d53c5b7924a03@mail.gmail.com>
In-Reply-To: <652776090902031140h370cb16fo600d53c5b7924a03@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


> Can you describe this in a little more detail; I'm not exactly sure what you
> mean.
>   

Break your large text documents into multiple Lucene documents. Rather 
than dividing them up into entirely discreet chunks of text consider 
storing/indexing *overlapping* sections of text with an overlap as big 
as the largest "slop" factor you use on Phrase/Span queries so that you 
don't cut any potential phrases in half and fail to match e.g.

This non-overlapping indexing scheme will not match a search for "George 
Bush"

    Doc 1 = "....  outgoing president George "
    Doc 2=  "Bush stated that ..."

While this overlapping scheme will match...
    Doc 1 = "....  outgoing president George "
    Doc 2=  "president George Bush stated that ..."

This fragmenting approach helps avoid the performance cost of 
highlighting very large documents.

The remaining issue is to remove duplicates in your search results when 
you match multiple chunks e.g. Lucene Docs #1 and #2 both refer to Input 
Doc#1 and will match a search for "president". You will need to store a 
field for the "original document number" and remove any duplicates (or 
merge them in the display if that is what is required).

Cheers,
Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org