Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Paul Libbrecht <paul@activemath.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: trying to use the highlighter
Date: Fri, 3 Sep 2010 14:05:53 +0200
Message-Id: <725AEC83-D03D-41D2-B3B7-5CF670809994@activemath.org>
To: java-user@lucene.apache.org
Mime-Version: 1.0 (Apple Message framework v1081)


Hello list,

I'm strugging again with the highlighter. I don't understand why I =
obtain sporadically InvalidTokenOffsetsException.

The mission: given a query, detect which field was matched, among the =
names of the concepts: there can be several names for a given concept, =
also in one language. Concepts are documents and names are in fields =
name-xx where xx is the two-letter-language.

Here's the method I'm using:

    public String computeMatchedField(int docNum, Document doc, Analyzer =
analyzer, Query query) throws IOException {
        //System.out.println("----- computing matched field for query " =
+ query + " on document " + doc.get("uri"));
        query =3D query.rewrite(this.reader);
        String found =3D null;
        float maxScore =3D 0;
        try {
            for(Field f: (List<Field>) doc.getFields()) {
                QueryScorer scorer =3D new =
QueryScorer(query,reader,f.name());
                if(!f.name().startsWith("name-")) continue;
                //System.out.println("Measuring field " + f.name() + ": =
" + f.stringValue());
                String text =3D f.stringValue();
                TokenStream tokenStream =3D =
TokenSources.getAnyTokenStream(reader,docNum, f.name(), doc, analyzer);
                SimpleHTMLFormatter htmlFormatter =3D new =
SimpleHTMLFormatter();
                Highlighter highlighter =3D new =
Highlighter(htmlFormatter, scorer);
                TextFragment[] frags =3D =
highlighter.getBestTextFragments(tokenStream, text, false, 1);
                if(frags=3D=3Dnull || frags.length=3D=3D0) continue;
                float score =3D frags[0].getScore();
                //System.out.println("Score: " + score);
                if(score > maxScore) {
                    maxScore =3D score;
                    found =3D frags[0].toString();
                }
            }
        } catch(Exception ex) {ex.printStackTrace();}
        return found;
    }

Unfortunately, I have to catch InvalidTokenOffsetsException which does =
happen sometimes, not always.
When it occurs, it stops the highlighting (the detected field is "null") =
and also costs quite some time.

What am I doing wrong?
I tried making my own tokenStream with no difference.

thanks in advance

paul


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org