Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Message-ID: <403D1E16.80704@apache.org>
Date: Wed, 25 Feb 2004 14:13:42 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040116
MIME-Version: 1.0
To: Lucene Developers List <lucene-dev@jakarta.apache.org>
Subject: Re: Dmitry's Term Vector stuff, plus some
References: <200402252159.i1PLx9sT006663@server0027.freedom2surf.net>
In-Reply-To: <200402252159.i1PLx9sT006663@server0027.freedom2surf.net>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

markharw00d@yahoo.co.uk wrote:
> Bruce,
> Could a short term ( and possibly compromised )solution to your performance problem be to offer only the first 3k of these large 200k docs to 
> the highlighter in order to minimize the amount of tokenization required? Arguably the most relevant bit of a document is typically in the first 1k anyway?

Or perhaps the highlighter could be changed to stop tokenizing a 
document after 1000 tokens when enough fragments have been found to 
produce a summary.  That way, if there are hits in the first part of the 
  document, which there probably usually are for high-scoring hits, then 
the time to compute the summary is bounded by something less than the 
document size.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org