lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr slow response collection-High Load
Date Fri, 09 Sep 2016 14:32:15 GMT
The soft commit interval governs opening new
searchers, which should be "warmed" in order
load up caches. Mu guess is that you're not doing much
warming and thus seeing long search times.

Most attachments are stripped by the mail server,
if you want people to see the images put them up somewhere
and provide a link is the usual practice.

Have you examined the options here?
https://wiki.apache.org/solr/SolrPerformanceFactors

Best,
Erick

On Fri, Sep 9, 2016 at 2:10 AM, kshitij tyagi
<kshitij.shopclues@gmail.com> wrote:
> Hi Ankush,
>
> As you are updating highly on one of the cores, hard commit will play a
> major role.
>
> Reason: During hard commits solr merges your segments and this is a time
> taking process.
>
> During merging of segments indexing of documents gets affected i.e. gets
> slower.
>
> Try figuring out the right number of segments you need to have and focus on
> analysing the merge process of solr when you are updating high amount of
> data.
>
> You will need to find the correct time for hard commits and the required
> number of segments for the collection.
>
> Hope this helps.
>
>
>
> On Fri, Sep 9, 2016 at 2:13 PM, Ankush Khanna <a.khanna@rebuy.com> wrote:
>
>> Hello,
>>
>> We are running some test for improving our solr performance.
>>
>> We have around 15 collections on our solr cluster.
>> But we are particularly interested in one collection holding high amount of
>> documents. (
>> https://gist.github.com/AnkushKhanna/9a472bccc02d9859fce07cb0204862da)
>>
>> Issue:
>> We see that there are high response time from the collection, for the same
>> queries, when user load or update load is increased.
>>
>> What are we aiming for:
>> Low response time (lower than 3 sec) in high update/traffic.
>>
>> Current collection, production:
>> * Solr Cloud, 2 Shards 2 Replicas
>> * Indexed: 5.4 million documents
>> * 45 indexed fields per document
>> * Soft commit: 5 seconds
>> * Hard commit: 10 minutes
>>
>> Test Setup:
>> * Indexed: 3 million documents
>> * Rest is same as in production
>> * Using gatling to mimic behaviour of updates and user traffic
>>
>> Finding:
>> We see the problem occurring more often when:
>> * query size is greater than 2000 characters (we can limit the search to
>> 2000 characters, but is there a solution to do this without limiting the
>> size)
>> * there is high updates going on
>> * high user traffic
>>
>> Some settings I explored:
>> * 1 Shard and 3 Replicas
>> * Hard commit: 5 minutes (Referencing
>> https://lucidworks.com/blog/2013/08/23/understanding-
>> transaction-logs-softcommit-and-commit-in-sorlcloud/
>> )
>>
>> With both the above solutions we see some improvements, but not drastic.
>> (Attach images)
>>
>> I would like to have more insights into the following questions:
>> * Why is there an improvement with lowering the hard commit time, would it
>> interesting to explore with lower hard commit time.
>>
>> Can some one provide some other pointer I could explore.
>>
>> Regards
>> Ankush Khanna
>>

Mime
View raw message