Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of markrmiller@gmail.com
 designates 64.233.184.235 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding;
        b=rXD6iC4ArEuHP7D/CbHRVlqKW2vniKBkWelwsUMSx0+nAvBbskF069jZOE5BBT26jPwtdJenYTQbkgwwE3ZMb67qKPjpmJhTTEloXeeyy6ZVzEfh57G7cfMt8CGVnWomVubprsZbC57MuvaXWetTpBBcQ2uUZtavhDDvoexTY5k=
Message-ID: <4679D058.9040403@gmail.com>
Date: Wed, 20 Jun 2007 21:11:52 -0400
From: Mark Miller <markrmiller@gmail.com>
User-Agent: Thunderbird 2.0.0.4 (Windows/20070604)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Highlighter that works with phrase and span queries
References: <280820.27409.qm@web50310.mail.re2.yahoo.com>
In-Reply-To: <280820.27409.qm@web50310.mail.re2.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I will work up some performance numbers over the next day or two to 
share with you. I have spent the last day or two with a profiler trying 
to find the biggest performance drains.

Unfortunately, I will probably not be able to squeeze out much more 
performance than the current Highlighter. When I started working on this 
project I considered starting from scratch to create a better, more 
accurate Highlighter. After some initial work I quickly came to the 
realization that Mark Harwood (with some additions by others) had 
already solved too many corner cases and interesting needs. The few 
alternate Highlighters in JIRA did not meet the standards set by Mark's 
highlighter. Trying to replicate all that work in a different manner 
didn't seem like a fruitful approach -- Harwood is more clever than I <g>

Taking that into account, I decided to extend the Highlighter using the 
great framework that is already in place. I implemented a new Scorer 
that acts much like the default Scorer, but when it finds a Query clause 
that is position sensitive (PhraseQuery, SpanQuery), it creates a 
MemoryIndex that is used extract the correct Spans for the Query (Credit 
to Paul Elschot and Mark Harwood for the approach). Non position 
sensitive Query claueses are handled similar to the way they where in 
the original highlighter's Scorer. This means that non position 
sensitive queries are likely the same speed as before, while position 
sensitive queries are likely a bit slower. For my uses, the thing is 
damned fast -- of course my uses involves small documents (Newspaper 
articles).

I am very interested in making this thing as fast as possible though, so 
I will build some benchmark tests and try to squeeze as much performance 
out of the Highligher as I can. I will also see if my Scorer is any 
faster than the original.

All that said, my guess is that one of the slowest parts of Highlighting 
is re-tokenizing the text. There is always the option of turning on 
TermVectors and using org.apache.lucene.search.highlight.TokenSources to 
get the TokenStream. Based on Mark H's comments, it may be twice as fast 
as re-tokenizing. This method can also be used with my new Highlighter 
code as well (which is more a plug-in to the old Highlighter than a 
replacement.)

Considering that both of your comments immediately went to performance, 
I will certainly be spending some time working on this.

- Mark

> Hi Mark,
>
> I know one large user (meaning: high query/highlight rates) of the current Highlighter and this user wasn't too happy with its performance.  I don't know the details, other than it was inefficient.  So now I'm wondering if you've benchmarked your Highlighter against that/current Highlighter to see not only which one is more accurate, but also which one is faster, and by how much?
>
> Thanks,
> Otis
>  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>   

> This is really great, Mark.  I'll look into integrating it with Solr,
> as better phrase highlighting is a definite sore point for some of our
> users.
>
>
>
> Any indication on performance differences?
>
>
>
> cheers,
>
> -mike
>
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org