lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nawab Zada Asad Iqbal <khi...@gmail.com>
Subject Re: Solr7: Bad query throughput around commit time
Date Sat, 11 Nov 2017 19:08:55 GMT
Thanks for a quick and detailed response, Erick!

Unfortunately i don't have a proof, but our servers with solr 4.5 are
running really nicely with the above config. I had assumed that same  or
similar settings will also perform well with Solr 7, but that assumption
didn't hold. As, a lot has changed in 3 major releases.
I have tweaked the cache values as you suggested but increasing or
decreasing doesn't seem to do any noticeable improvement.

At the moment, my one core has 800GB index, ~450 Million documents, 48 G
Xmx. GC pauses haven't been an issue though.  One machine runs with a 3TB
drive, running 3 solr processes (each with one core as described above).  I
agree that it is a very atypical system so i should probably try different
parameters with a fresh eye to find the solution.


I tried with autocommits (commit with opensearcher=false very half minute ;
and softcommit every 5 minutes). That supported the hypothesis that the
query throughput decreases after opening a new searcher and **not** after
committing the index . Cache hit ratios are all in 80+% (even when i
decreased the filterCache to 128, so i will keep it at this lower value).
Document cache hitratio is really bad, it drops to around 40% after
newSearcher. But i guess that is expected, since it cannot be warmed up
anyway.


Thanks
Nawab



On Thu, Nov 9, 2017 at 9:11 PM, Erick Erickson <erickerickson@gmail.com>
wrote:

> What evidence to you have that the changes you've made to your configs
> are useful? There's lots of things in here that are suspect:
>
>   <double name="forceMergeDeletesPctAllowed">1</double>
>
> First, this is useless unless you are forceMerging/optimizing. Which
> you shouldn't be doing under most circumstances. And you're going to
> be rewriting a lot of data every time See:
>
> https://lucidworks.com/2017/10/13/segment-merging-deleted-
> documents-optimize-may-bad/
>
> filterCache size of size="10240" is far in excess of what we usually
> recommend. Each entry can be up to maxDoc/8 and you have 10K of them.
> Why did you choose this? On the theory that "more is better?" If
> you're using NOW then you may not be using the filterCache well, see:
>
> https://lucidworks.com/2012/02/23/date-math-now-and-filter-queries/
>
> autowarmCount="1024"
>
> Every time you commit you're firing off 1024 queries which is going to
> spike the CPU a lot. Again, this is super-excessive. I usually start
> with 16 or so.
>
> Why are you committing from a cron job? Why not just set your
> autocommit settings and forget about it? That's what they're for.
>
> Your queryResultCache is likewise kind of large, but it takes up much
> less space than the filterCache per entry so it's probably OK. I'd
> still shrink it and set the autowarm to 16 or so to start, unless
> you're seeing a pretty high hit ratio, which is pretty unusual but
> does happen.
>
> 48G of memory is just asking for long GC pauses. How many docs do you
> have in each core anyway? If you're really using this much heap, then
> it'd be good to see what you can do to shrink in. Enabling docValues
> for all fields you facet, sort or group on will help that a lot if you
> haven't already.
>
> How much memory on your entire machine? And how much is used by _all_
> the JVMs you running on a particular machine? MMapDirectory needs as
> much OS memory space as it can get, see:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Lately we've seen some structures that consume memory until a commit
> happens (either soft or hard). I'd shrink my autocommit down to 60
> seconds or even less (openSearcher=false).
>
> In short, I'd go back mostly to the default settings and build _up_ as
> you can demonstrate improvements. You've changed enough things here
> that untangling which one is the culprit will be hard. You want the
> JVM to have as little memory as possible, unfortunately that's
> something you figure out by experimentation.
>
> Best,
> Erick
>
> On Thu, Nov 9, 2017 at 8:42 PM, Nawab Zada Asad Iqbal <khichi@gmail.com>
> wrote:
> > Hi,
> >
> > I am committing every 5 minutes using a periodic cron job  "curl
> > http://localhost:8984/solr/core1/update?commit=true". Besides this, my
> app
> > doesn't do any soft or hard commits. With Solr 7 upgrade, I am noticing
> > that query throughput plummets every 5 minutes - probably when the commit
> > happens.
> > What can I do to improve this? I didn't use to happen like this in
> solr4.5.
> > (i.e., i used to get a stable query throughput of  50-60 queries per
> > second. Now there are spikes to 60 qps interleaved by drops to almost
> > **0**).  Between those 5 minutes, I am able to achieve high throughput,
> > hence I guess that issue is related to indexing or merging, and not query
> > flow.
> >
> > I have 48G allotted to each solr process, and it seems that only ~50% is
> > being used at any time, similarly CPU is not spiking beyond 50% either.
> > There is frequent merging (every 5 minute) , but i am not sure if that is
> > a cause of the slowdown.
> >
> > Here are my merge and cache settings:
> >
> > Thanks
> > Nawab
> >
> > <mergePolicyFactory class="org.apache.solr.index.
> TieredMergePolicyFactory">
> >   <int name="maxMergeAtOnce">5</int>
> >   <int name="segmentsPerTier">5</int>
> >       <int name="maxMergeAtOnceExplicit">10</int>
> >       <int name="floorSegmentMB">16</int>
> >       <!-- 50 gb -->
> >       <double name="maxMergedSegmentMB">50000</double>
> >       <double name="forceMergeDeletesPctAllowed">1</double>
> >
> >     </mergePolicyFactory>
> >
> >
> >
> >
> > <filterCache class="solr.FastLRUCache"
> >              size="10240"
> >              initialSize="5120"
> >              autowarmCount="1024"/>
> > <queryResultCache class="solr.LRUCache"
> >                  size="10240"
> >                  initialSize="5120"
> >                  autowarmCount="0"/>
> > <documentCache class="solr.LRUCache"
> >                size="10240"
> >                initialSize="5120"
> >                autowarmCount="0"/>
> >
> >
> > <useColdSearcher>false</useColdSearcher>
> >
> > <maxWarmingSearchers>2</maxWarmingSearchers>
> >
> > <listener event="newSearcher" class="solr.QuerySenderListener">
> >   <arr name="queries">
> >   </arr>
> > </listener>
> > <listener event="firstSearcher" class="solr.QuerySenderListener">
> >   <arr name="queries">
> >   </arr>
> > </listener>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message