Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of jake.mannix@gmail.com
 designates 209.85.211.184 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=tz/8v89vanTdMNDCd73i1f03WU/0toKzLtCwoD24xwNgzaY0stx3I5B80UkUQnfLmO
         xs7neXjwhEIcx5H5rsp6e7EuQyLqpKBZlKq07kDjF40Ya4MgKd92mQqtNwSBbZfQ2vTj
         vpRbUIjgwxBk9t7a2RhQ/p2IcaZp5VzMc1M0A=
MIME-Version: 1.0
In-Reply-To: <8669da1e0910121030w5ad239d9u52b447ff89b4f87@mail.gmail.com>
References: <8669da1e0910120553t28efc634t9f808c3675021e53@mail.gmail.com>
	 <200910121745.12793.paul.elschot@xs4all.nl>
	 <8837fb770910120932r21555aa0t59e9b36db5663f70@mail.gmail.com>
	 <8669da1e0910121030w5ad239d9u52b447ff89b4f87@mail.gmail.com>
Date: Mon, 12 Oct 2009 11:02:02 -0700
Message-ID: <4b124c310910121102k5fe69191r5a3ac0c5c4838bbd@mail.gmail.com>
Subject: Re: faceted search performance
From: Jake Mannix <jake.mannix@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016e646182e13e1790475c0baa1

--0016e646182e13e1790475c0baa1
Content-Type: text/plain; charset=ISO-8859-1

Hey Chris,

On Mon, Oct 12, 2009 at 10:30 AM, Christoph Boosz <
christoph.boosz@googlemail.com> wrote:

> Thanks for your reply.
> Yes, it's likely that many terms occur in few documents.
>
> If I understand you right, I should do the following:
> -Write a HitCollector that simply increments a counter
> -Get the filter for the user query once: new CachingWrapperFilter(new
> QueryWrapperFilter(userQuery));
> -Create a TermQuery for each term
> -Perform the search and read the counter of the HitCollector
>
> I did that, but it didn't get faster. Any ideas why?
>

This killer is the "TermQuery for each term" part - this is huge. You need
to invert this process,
and use your query as is, but while walking in the HitCollector, on each doc
which matches
your query, increment counters for each of the terms in that document (which
means you need
an in-memory forward lookup for your documents, like a multivalued
FieldCache - and if you've
got roughly the same number of terms as documents, this cache is likely to
be as large as
your entire index - a pretty hefty RAM cost).

But a good thing to keep in mind is that doing this kind of faceting
(massively multivalued
on a huge term-set) requires a lot of computation, even if you have all the
proper structures
living in memory:

For each document you look at (which matches your query), you need to look
at all
of the terms in that document, and increment a counter for that term.  So
however much
time it would normally take for you to do the driving query, it can take as
much as that
multiplied by the average number of terms in a document in your index.  If
your documents
are big, this could be a pretty huge latency penalty.

  -jake

--0016e646182e13e1790475c0baa1--