Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Thu, 12 May 2011 15:58:47 +0000 (UTC)
From: =?utf-8?Q?Pierre_Goss=C3=A9_=28JIRA=29?= <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <1476424519.7056.1305215927450.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1040365464.3003.1305121667325.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (LUCENE-3087) highlighting exact phrase with
 overlapping tokens fails.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/LUCENE-3087?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
32472#comment-13032472 ]=20

Pierre Goss=C3=A9 commented on LUCENE-3087:
--------------------------------------

Yes, that would be the best.

But I'm not sure how to do that :
- Check for positions in token stream ? Not sure it "guaranties" anything. =
:)
- add some kind of additionnal properties to the tokenstream returned by th=
e IndexReader.getTermFreqVector since it access fields info ? Not sure it h=
as'nt too much impact.
- Ask the index for field infos from TokenSources.getTokenStream ? Not sure=
 it is the place but look like the less dangerous option.

I haven't much time 'till the end of month to take a serious look at this, =
but I'll try to take some time next month.

> highlighting exact phrase with overlapping tokens fails.
> --------------------------------------------------------
>
>                 Key: LUCENE-3087
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3087
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: contrib/highlighter
>    Affects Versions: 2.9.4, 3.1
>            Reporter: Pierre Goss=C3=A9
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.2, 4.0
>
>         Attachments: LUCENE-3087.patch
>
>
> Fields with overlapping token are not highlighted in search results when =
searching exact phrases, when using TermVector.WITH_OFFSET.
> The document builded in MemoryIndex for highlight does not preserve posit=
ions of tokens in this case. Overlapping tokens get "flattened" (position i=
ncrement always set to 1), the spanquery used for searching relevant fragme=
nt will fail to identify the correct token sequence because the position sh=
ift.
> I corrected this by adding a position increment calculation in sub class =
StoredTokenStream. I added junit test covering this case.
> I used the eclipse codestyle from trunk, but style add quite a few format=
 differences between repository and working copy files. I tried to reduce t=
hem, but some linewrapping rules still doesn't match.
> Correction patch joined

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org