lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Benedetti <abenede...@apache.org>
Subject Re: [Faceting] Exact Value Faceting VS ID Faceting
Date Thu, 26 Nov 2015 10:07:14 GMT
Thanks Toke for the answer, let me comment inline :

On 26 November 2015 at 08:32, Toke Eskildsen <te@statsbiblioteket.dk> wrote:

> On Wed, 2015-11-25 at 15:56 +0000, Alessandro Benedetti wrote:
> > I would like to have docValues because facets are going to be heavy on
> > those fields.
>
> > *Faceting approach *
> > *1) *Indexing the human readable field value
>
> Technically this will be a SORTED or SORTED_SET, which again means that
> a pool of terms is maintained for each segment. The mapping from
> documents to terms are done using ordinals, which are not comparable
> across segments.
>

Thank you very much, I missed that part in my initial analysis.
I overlooked the fact that a segment is a fully working Lucene index, and
actually they are independent each others ( in term dictionary for example)
.
So the ordinal resolution is absolutely something to consider.


>
> > Facets will be returned readable, out of the box.
> > I can not see any cons in this approach, I would say it is the standard
> one.
>
> With multiple segments, the terms from each segment must somehow be
> aligned, do avoid duplicate entries in the result. This can either be
> done be creating a segment_ordinal->global_ordinal map upon first
> faceting call (facet.method=fc) or by on-the-fly comparison of top-X
> terms from each segment (facet.method=fcs). Either way, there is a
> performance penalty.
>
> >    - When calculating faceting, in memory it is used the ordinal for each
> >    term, which means in memory we don't waste space for the actual term,
> or
> >    waste the time looking up for the value until the very end of the
> process,
> >    after the counts are done .
>
> The segment_ordinal->global_ordinal requires memory linear to the number
> of unique values in the field. If fcs is used, there will be more term
> lookups.
>
> > *2)* Correlate outside the search system each term to a custom ID. Index
> > the custom ID. After facets are calculated resolve the ID and show the
> > human readable labels.
>
> Assuming the ID is an integer (about the only thing that makes sense),
> this ensures that the IDs are comparable across segments, so no
> segment->global mapping is needed. This removes the performance penalty
> described above and is (as far as I understand) the principle behind
> Lucene faceting.
>

Ok, so in the case of Integer faceting, we don't do the ordinal resolution
and we count directly the integer values , right ?

>
> On the other hand, this approach is indeed more complicated and it
> introduces another hotspot both for indexing (as document construction
> requires a lookup in the term provider) and searching (for resolving the
> final terms).
>
> I agree.


>
>
> If we had a hashing method String->long and guaranteed that there would
> be no collisions (or we accepted the occasional faulty result), then we
> could avoid the segment->global map as well as the centralized term
> server. To my knowledge, this has not yet been attempted.
>
>
Thank you very much !


> - Toke Eskildsen
>
>
>


-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message