lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <yo...@apache.org>
Subject Re: Facet performance with heterogeneous 'facets'?
Date Tue, 19 Sep 2006 13:51:19 GMT
On 9/18/06, Michael Imbeault <michael.imbeault@sympatico.ca> wrote:
> Yonik Seeley wrote:
> > For cases like "author", if there is only one value per document, then
> > a possible fix is to use the field cache.  If there can be multiple
> > occurrences, there doesn't seem to be a good way that preserves exact
> > counts, except maybe if the number of documents matching a query is
> > low.
> >
> I have one value per document (I have fields for authors, last_author
> and first_author, and I'm doing faceted search on first and last authors
> fields). How would I use the field cache to fix my problem?

Unless you want to dive into Solr development, you don't :-)
It requires extensive changes to the faceting code and doing things a
different way in some cases.

The FieldCache is the fastest way to "uninvert" single valued
fields... it's currently only used for Sorting, where one needs to
quickly know the field value given the document id.
The downside is high memory use, and that it's not a general
solution... it can't handle fields with multiple tokens (tokenized
fields or multi-valued fields).

So the strategy would be to step through the documents, get the value
for the field from the FieldCache, increment a counter for that value,
then find the top counters when we are done.

> Also, would
> it be better to store a unique number (for each possible author) in an
> int field along with the string, and do the faceted searching on the int
> field?

It won't really help.  It wouldn't be faster, and it would require
only slightly less memory.

> >> Just a little follow-up - I did a little more testing, and the query
> >> takes 20 seconds no matter what - If there's one document in the results
> >> set, or if I do a query that returns all 130000 documents.
> >
> > Yes, currently the same strategy is always used.
> >   intersection_count(docs_matching_query, docs_matching_author1)
> >   intersection_count(docs_matching_query, docs_matching_author2)
> >   intersection_count(docs_matching_query, docs_matching_author3)
> >   etc...
> >
> > Normally, the docsets will be cached, but since the number of authors
> > is greater than the size of the filtercache, the effective cache hit
> > rate will be 0%
> >
> > -Yonik
> So more memory would fix the problem?

Yes, if your collection size isn't that large...  it's not a practical
solution for many cases though.

> Also, I was under the impression
> that it was only searching / sorting for authors that it knows are in
> the result set...

That's the problem... it's not necessarily easy to know *what* authors
are in the result set.  If we could quickly determine that, we could
just count them and not do any intersections or anything at all.

>  in the case of only one document (1 result), it seems
> strange that it takes the same time as for 130 000 results. It should
> just check the results, see that there's only one author, and return
> that? And in the case of 2 documents, just sort 2 authors (or 1 if
> they're the same)? I understand your answer (it does intersections), but
> I wonder why its intersecting from the whole document set at first, and
> not docs_matching_query like you said.

It is just intersecting docs_matching_query.  The problem is that it's
intersecting that set with all possible author sets since it doesn't
know ahead of time what authors are in the docs that match the query.

There could be optimizations when docs_matching_query.size() is small,
so we start somehow with terms in the documents rather than terms in
the index.  That requires termvectors to be stored (medium speed), or
requires that the field be stored and that we re-analyze it (very
slow).

More optimization of special cases hasn't been done simply because no
one has done it yet... (as you note, faceting is a new feature).


-Yonik

Mime
View raw message