lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankush Khanna <a.kha...@rebuy.com>
Subject Re: Solr slow response collection-High Load
Date Fri, 09 Sep 2016 16:33:30 GMT
Thanks Erick and Kshitij.
Would try both the options and see what works best.

Regards
Ankush Khanna

On Fri, 9 Sep 2016 at 16:33 Erick Erickson <erickerickson@gmail.com> wrote:

> The soft commit interval governs opening new
> searchers, which should be "warmed" in order
> load up caches. Mu guess is that you're not doing much
> warming and thus seeing long search times.
>
> Most attachments are stripped by the mail server,
> if you want people to see the images put them up somewhere
> and provide a link is the usual practice.
>
> Have you examined the options here?
> https://wiki.apache.org/solr/SolrPerformanceFactors
>
> Best,
> Erick
>
> On Fri, Sep 9, 2016 at 2:10 AM, kshitij tyagi
> <kshitij.shopclues@gmail.com> wrote:
> > Hi Ankush,
> >
> > As you are updating highly on one of the cores, hard commit will play a
> > major role.
> >
> > Reason: During hard commits solr merges your segments and this is a time
> > taking process.
> >
> > During merging of segments indexing of documents gets affected i.e. gets
> > slower.
> >
> > Try figuring out the right number of segments you need to have and focus
> on
> > analysing the merge process of solr when you are updating high amount of
> > data.
> >
> > You will need to find the correct time for hard commits and the required
> > number of segments for the collection.
> >
> > Hope this helps.
> >
> >
> >
> > On Fri, Sep 9, 2016 at 2:13 PM, Ankush Khanna <a.khanna@rebuy.com>
> wrote:
> >
> >> Hello,
> >>
> >> We are running some test for improving our solr performance.
> >>
> >> We have around 15 collections on our solr cluster.
> >> But we are particularly interested in one collection holding high
> amount of
> >> documents. (
> >> https://gist.github.com/AnkushKhanna/9a472bccc02d9859fce07cb0204862da)
> >>
> >> Issue:
> >> We see that there are high response time from the collection, for the
> same
> >> queries, when user load or update load is increased.
> >>
> >> What are we aiming for:
> >> Low response time (lower than 3 sec) in high update/traffic.
> >>
> >> Current collection, production:
> >> * Solr Cloud, 2 Shards 2 Replicas
> >> * Indexed: 5.4 million documents
> >> * 45 indexed fields per document
> >> * Soft commit: 5 seconds
> >> * Hard commit: 10 minutes
> >>
> >> Test Setup:
> >> * Indexed: 3 million documents
> >> * Rest is same as in production
> >> * Using gatling to mimic behaviour of updates and user traffic
> >>
> >> Finding:
> >> We see the problem occurring more often when:
> >> * query size is greater than 2000 characters (we can limit the search to
> >> 2000 characters, but is there a solution to do this without limiting the
> >> size)
> >> * there is high updates going on
> >> * high user traffic
> >>
> >> Some settings I explored:
> >> * 1 Shard and 3 Replicas
> >> * Hard commit: 5 minutes (Referencing
> >> https://lucidworks.com/blog/2013/08/23/understanding-
> >> transaction-logs-softcommit-and-commit-in-sorlcloud/
> >> )
> >>
> >> With both the above solutions we see some improvements, but not drastic.
> >> (Attach images)
> >>
> >> I would like to have more insights into the following questions:
> >> * Why is there an improvement with lowering the hard commit time, would
> it
> >> interesting to explore with lower hard commit time.
> >>
> >> Can some one provide some other pointer I could explore.
> >>
> >> Regards
> >> Ankush Khanna
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message