Mailing-List: contact lucy-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Thu, 10 Nov 2011 14:53:19 -0800
From: Marvin Humphrey <marvin@rectangular.com>
To: lucy-user@incubator.apache.org
Message-ID: <20111110225319.GA25189@rectangular.com>
References: 
 <CAKn6EF_YYz63+jNJkqW2wiVB8G=NtPjH+tg_et_o=eBBk+iQXA@mail.gmail.com>
 <20111109160921.GA8356@rectangular.com>
 <CAKn6EF-R_5DksnKqf1W+0e8h55VhA0JQMsUwqesz8uxpc98Hwg@mail.gmail.com>
 <CAKn6EF_BdNPS4xnzOnBnCaxAaYs2izo6OcRCHBXQ5pCJBgaDLw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAKn6EF_BdNPS4xnzOnBnCaxAaYs2izo6OcRCHBXQ5pCJBgaDLw@mail.gmail.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
Subject: Re: [lucy-user] Highlighting problem with latest trunk

On Thu, Nov 10, 2011 at 11:31:31AM +0200, goran kent wrote:
> On 11/10/11, goran kent <gorankent@gmail.com> wrote:
> > Completing each new() requires 4s *each*.  Somehow I don't recall this
> > being the case before :/
> 
> The 4s x 2 penalty is probably related to the remote searching (of
> which highlighting is a part) doing things serially and not
> concurrently, and not this particular bug, no?

Correct, that is the most important factor.  There are likely a lot of remote
doc_freq() calls bouncing around, and those calls are being executed serially
by PolySearcher.

Highlighter requires a weighted query -- in Lucy-speak, a
Lucy::Search::Compiler[1] -- in order to determine both which parts of the
field matched and how much they contributed to the score.  In order to weight
a query, you need to know how common each term is, so that in a search for
'the metamorphosis' the term 'metamorphosis' contributes more to the score
than the term 'the'.  In the context of Highlighter, we need to know that
'metamorphosis' is more important than 'the' so that we can prefer
selections which contain 'metamorphosis' over selections which contain 'the'
when choosing an excerpt.

However, having Highlighter weight the query means duplicated work, because
the main Searcher has to perform exactly the same weighting routine prior to
asking the remote nodes to score results.

We can eliminate the duplicated effort by performing the weighting manually
and supplying the weighted query object to both Searcher#hits and
Highlighter#new.  (You'll need the latest trunk because this sample code
requires a patch from LUCY-188 I committed earlier today.)

    my $query    = $query_parser->parse($query_string);
    my $compiler = $query->make_compiler(searcher => $searcher);
    my $hits     = $searcher->hits(query => $compiler);
    my $highlighter = Lucy::Highlight::Highlighter->new(
        query    => $compiler,
        searcher => $searcher,
        field    => 'content',
    );

You may be able to cut down those remote doc_freq() calls further by using a
QueryParser which has the minimum possible number of fields.  I can go into
depth on that in another email if you like.

Marvin Humphrey

[1] It's called a "Compiler" because it's primary role is to compile a Query
    to a Matcher.  Nobody likes the name, but we haven't achieved consensus on
    what to do about it.