lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven White <swhite4...@gmail.com>
Subject Re: Number of fields in qf & fq
Date Fri, 20 Nov 2015 13:08:46 GMT
Thanks Erick.

The 1500 fields is a design that I inherited.  I'm trying to figure out why
it was done as such and what it will take to fix it.

What about my other question: how does one go about debugging performance
issues in Solr to find out where time is mostly spent?  How do I know my
Solr parameters, such as cache and what have you are set right?  From what
I see, we are using the defaults off solrconfig.xml.

I'm on Solr 5.2

Steve


On Thu, Nov 19, 2015 at 11:36 PM, Erick Erickson <erickerickson@gmail.com>
wrote:

> An fq is still a single entry in your filterCache so from that
> perspective it's the same.
>
> And to create that entry, you're still using all the underlying fields
> to search, so they have to be loaded just like they would be in a q
> clause.
>
> But really, the fundamental question here is why your design even has
> 1,500 fields and, more specifically, why you would want to search them
> all at once. From a 10,000 ft. view, that's a very suspect design.
>
> Best,
> Erick
>
> On Thu, Nov 19, 2015 at 4:06 PM, Walter Underwood <wunder@wunderwood.org>
> wrote:
> > The implementation for fq has changed from 4.x to 5.x, so I’ll let
> someone else answer that in detail.
> >
> > In 4.x, the result of each filter query can be cached. After that, they
> are quite fast.
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >
> >> On Nov 19, 2015, at 3:59 PM, Steven White <swhite4141@gmail.com> wrote:
> >>
> >> Thanks Walter.  I see your point.  Does this apply to fq as will?
> >>
> >> Also, how does one go about debugging performance issues in Solr to find
> >> out where time is mostly spent?
> >>
> >> Steve
> >>
> >> On Thu, Nov 19, 2015 at 6:54 PM, Walter Underwood <
> wunder@wunderwood.org>
> >> wrote:
> >>
> >>> With one field in qf for a single-term query, Solr is fetching one
> posting
> >>> list. With 1500 fields, it is fetching 1500 posting lists. It could
> easily
> >>> be 1500 times slower.
> >>>
> >>> It might be even slower than that, because we can’t guarantee that: a)
> >>> every algorithm in Solr is linear, b) that all those lists will fit in
> >>> memory.
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wunder@wunderwood.org
> >>> http://observer.wunderwood.org/  (my blog)
> >>>
> >>>
> >>>> On Nov 19, 2015, at 3:46 PM, Steven White <swhite4141@gmail.com>
> wrote:
> >>>>
> >>>> Hi everyone
> >>>>
> >>>> What is considered too many fields for qf and fq?  On average I will
> have
> >>>> 1500 fields in qf and 100 in fq (all of which are OR'ed).  Assuming
I
> can
> >>>> (I have to check with the design) for qf, if I cut it down to 1 field,
> >>> will
> >>>> I see noticeable performance improvement?  It will take a lot of
> effort
> >>> to
> >>>> test this which is why I'm asking first.
> >>>>
> >>>> As is, I'm seeing 2-5 sec response time for searches on an index of
1
> >>>> million records with total index size (on disk) of 4 GB.  I gave Solr
> 2
> >>> GB
> >>>> of RAM (also tested at 4 GB) in both cases Solr didn't use more then
1
> >>> GB.
> >>>>
> >>>> Thanks in advanced
> >>>>
> >>>> Steve
> >>>
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message