Mailing-List: contact lucy-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of gorankent@gmail.com designates
 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <20111110225319.GA25189@rectangular.com>
References: 
 <CAKn6EF_YYz63+jNJkqW2wiVB8G=NtPjH+tg_et_o=eBBk+iQXA@mail.gmail.com>
	<20111109160921.GA8356@rectangular.com>
	<CAKn6EF-R_5DksnKqf1W+0e8h55VhA0JQMsUwqesz8uxpc98Hwg@mail.gmail.com>
	<CAKn6EF_BdNPS4xnzOnBnCaxAaYs2izo6OcRCHBXQ5pCJBgaDLw@mail.gmail.com>
	<20111110225319.GA25189@rectangular.com>
Date: Fri, 11 Nov 2011 09:04:38 +0200
Message-ID: 
 <CAKn6EF8DRejBFjNKhZjgK5JKidT932O3_uSJ=q9ODeNUK9_jMw@mail.gmail.com>
From: goran kent <gorankent@gmail.com>
To: lucy-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: Re: [lucy-user] Highlighting problem with latest trunk

On Fri, Nov 11, 2011 at 12:53 AM, Marvin Humphrey
<marvin@rectangular.com> wrote:
> Correct, that is the most important factor. =A0There are likely a lot of =
remote
> doc_freq() calls bouncing around, and those calls are being executed seri=
ally
> by PolySearcher.
>
> Highlighter requires a weighted query -- in Lucy-speak, a
> Lucy::Search::Compiler[1] -- in order to determine both which parts of th=
e
> field matched and how much they contributed to the score. =A0In order to =
weight
> a query, you need to know how common each term is, so that in a search fo=
r
> 'the metamorphosis' the term 'metamorphosis' contributes more to the scor=
e
> than the term 'the'. =A0In the context of Highlighter, we need to know th=
at
> 'metamorphosis' is more important than 'the' so that we can prefer
> selections which contain 'metamorphosis' over selections which contain 't=
he'
> when choosing an excerpt.
>
> However, having Highlighter weight the query means duplicated work, becau=
se
> the main Searcher has to perform exactly the same weighting routine prior=
 to
> asking the remote nodes to score results.
>
> We can eliminate the duplicated effort by performing the weighting manual=
ly
> and supplying the weighted query object to both Searcher#hits and
> Highlighter#new. =A0(You'll need the latest trunk because this sample cod=
e
> requires a patch from LUCY-188 I committed earlier today.)
>
> =A0 =A0my $query =A0 =A0=3D $query_parser->parse($query_string);
> =A0 =A0my $compiler =3D $query->make_compiler(searcher =3D> $searcher);
> =A0 =A0my $hits =A0 =A0 =3D $searcher->hits(query =3D> $compiler);
> =A0 =A0my $highlighter =3D Lucy::Highlight::Highlighter->new(
> =A0 =A0 =A0 =A0query =A0 =A0=3D> $compiler,
> =A0 =A0 =A0 =A0searcher =3D> $searcher,
> =A0 =A0 =A0 =A0field =A0 =A0=3D> 'content',
> =A0 =A0);
>
> You may be able to cut down those remote doc_freq() calls further by usin=
g a
> QueryParser which has the minimum possible number of fields. =A0I can go =
into
> depth on that in another email if you like.
>
> Marvin Humphrey
>
> [1] It's called a "Compiler" because it's primary role is to compile a Qu=
ery
> =A0 =A0to a Matcher. =A0Nobody likes the name, but we haven't achieved co=
nsensus on
> =A0 =A0what to do about it.

Crikey, I'm so accustomed to the oftentimes terse replies from FOSS
devs that receiving such detailed and obviously
time-consuming-to-compile replies leaves me staring at the headlights
dumbly;  and I dribbled a bit there.

ok, back to Lucy - that would explain the herds of mysterious
doc_freq's I was seeing in my remote debug prints.  At the time, it
was like, "what the hell?"  The dim light of comprehension blinks on,
well, dimly.

Let me chew on this stuff a bit and I'll report back with some results.