lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: Huge FacetArrays while using SortedSetDocValuesAccumulator
Date Wed, 28 Aug 2013 15:26:25 GMT
Oops you're right, it was committed in LUCENE-4985 which will be released
in Lucene 4.5.

Shai


On Wed, Aug 28, 2013 at 6:16 PM, Krishnamurthy, Kannan <
Kannan.Krishnamurthy@contractor.cengage.com> wrote:

> Thanks for the response. I double checked that
> SortedSetDocValuesAccumulator doesn't take a FacetArray in its ctor
> currently in 4.3.0 and 4.4. But FacetAccumulator does take FacetArray in
> its ctor. Am I missing something here? We have a high traffic application
> currently doing about 250 searches and facet request per second. We haven't
> performance tested our facet implementation yet to see if object allocation
> is a problem.
>
> Thanks,
> +Kannan.
>
>
>
> Hi
>
> SortedSetDocValuesAccumulator does receive FacetArrays in its ctor, so you
> can pass ReusingFacetArrays. You will need to call FacetArrays.free() when
> you're done with accumulation though. However, do notice that
> ReusingFacetArrays did not show any big gain even with large taxonomies --
> that is that the overhead of allocating and freeing them wasn't noticeable.
>
> If you expect to use very large taxonomies, then facet partitions can help.
> But for that you need to use the sidecar taxonomy index.
>
> Shai
>
>
> On Mon, Aug 26, 2013 at 11:45 PM, Krishnamurthy, Kannan <
> Kannan.Krishnamurthy@contractor.cengage.com> wrote:
>
> > Hello,
> >
> > We are working with large lucene 4.3.0 index and using
> > SortedSetDocValuesFacetFields for creating facets and
> > SortedSetDocValuesAccumulator for facet accumulation. We couldn't use a
> > taxonomy based facet implementation (We use MultiReader for searching and
> > our indices is composed of multiple physical lucene indices, hence we
> > cannot have a single taxonomy index). We have two million categories and
> > expect to have another two million in the near future. As the current
> > implementation of SortedSetDocValuesAccumulator does not support
> > ReusingFacetArrays, we are concerned with potential garabage collector
> > related performance issues in our high traffic application. Will future
> > Lucene release support using ReusingFacetArrays in
> > SortedSetDocValuesAccumulator ?
> >
> > Also as an alternative we are considering subclassing FacetIndexingParams
> > and provide dimension specific CategoryListParams during indexing time.
> > This will help to reduce the size of the FacetArray per facet request. We
> > realize this approach will not support multiple FacetRequest in a single
> > SortedSetDocValuesAccumulator, as SortedSetDocValuesReaderState hardcodes
> > the category to null while calling
> > FacetIndexingParams.getCategoryListParams(null) in its constructor.
> >
> > Are there better approaches to this problem ?
> >
> >
> > Thanks in advance for any help.
> >
> > Kannan
> > Cengage Learning
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message