Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
Mime-Version: 1.0 (Apple Message framework v746.2)
In-Reply-To: <20051208155057.F3EE69ED04@mail-in-07.arcor-online.net>
References: <20051208155057.F3EE69ED04@mail-in-07.arcor-online.net>
Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed
Message-Id: <182A9813-DA9E-45DB-9306-B3CFE9210DCD@ehatchersolutions.com>
Content-Transfer-Encoding: quoted-printable
From: Erik Hatcher <erik@ehatchersolutions.com>
Subject: Re: pdf and highlighting
Date: Thu, 8 Dec 2005 15:07:38 -0500
To: java-user@lucene.apache.org


On Dec 8, 2005, at 10:51 AM, Sonja L=F6hr wrote:
> Thank you both, I found it
> (I really asked a bit too early, sorry)
>
> The highlighter works correct if I use my custom Analyzer during =20
> indexing
> (and for QueryParser), BUT
> when preparing the TokenStream to feed the highlighter, I must NOT =20
> use it.
>
> TokenStream tStream =3D new GermanAnalyzer().tokenStream("body", new
> StringReader(bodyText));	=09
> System.out.println( highlighter.getBestFragments(tStream, bodyText, =20=

> 4, "
> ..... "));
>
> works, wheras
>
> TokenStream tStream =3D new GermanHtmlAnalyzer().tokenStream("body", =
new
> StringReader(bodyText));	=09
> System.out.println( highlighter.getBestFragments(tStream, bodyText, =20=

> 4, "
> ..... "));
>
> gives rubbish highlighting.
>
> GermanHtmlAnalyzer feeds a normal GermanAnalyzer with a shortened =20
> String
> (native characters) if the input contains decimal or html entities, =20=

> but then
> I'm totally confused why there is a problem with pdf text and not =20
> with HTML
> text...

The likely reason is that the token offsets fed to the highlighter =20
don't jive with the positions of the text in the text you're =20
highlighting.  You're generating token offsets for strings that have =20
been replaced (and likely different sizes), but highlighting the =20
original text with the entities left intact.

Maybe??

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org