lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Heisey (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-4509) Disable Stale Check - Distributed Search (Performance)
Date Wed, 27 Feb 2013 18:45:13 GMT

    [ https://issues.apache.org/jira/browse/SOLR-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588597#comment-13588597
] 

Shawn Heisey edited comment on SOLR-4509 at 2/27/13 6:43 PM:
-------------------------------------------------------------

Interested user here, not anyone linked to Solr or Lucene.

Would the 100ms latency you have described here show up in QTime values or be purely client-side?
 I ask because the median (not average) QTime value I see in disributed searches (seven shards
on two servers) on Solr 3.5.0 is about 10 milliseconds, and even faster on 4.2-SNAPSHOT ...
and that involves the stale check both on the client side and the seven shards.  The client
is SolrJ 4.1, with max retries (1) and connection timeout (5000ms) being the only low-level
parameters that are set.  SolrCloud is not involved here.

The idea of a sweeper thread makes me nervous, given the project's general reaction to the
background threads created by an intermediate SOLR-1972 patch.  Also, with 5000 milliseconds
between executions of the sweeper thread and an assumption of a server timeout of 50 milliseconds,
will it be able to effectively avoid problems in all environments?  If these are not actual
worries, then I can be quiet.

                
      was (Author: elyograg):
    Interested user here, not anyone linked to Solr or Lucene.

Would the 100ms latency you have described here show up in QTime values or be purely client-side?
 I ask because the median (not average) QTime value I see in disributed searches (seven shards
on two servers) on Solr 3.5.0 is about 10 milliseconds, and even faster on 4.2-SNAPSHOT ...
and that involves the stale check both on the client side and the seven shards.  The client
is SolrJ 4.1, with max retries (2) and connection timeout (5000ms) being the only low-level
parameters that are set.  SolrCloud is not involved here.

The idea of a sweeper thread makes me nervous, given the project's general reaction to the
background threads created by an intermediate SOLR-1972 patch.  Also, with 5000 milliseconds
between executions of the sweeper thread and an assumption of a server timeout of 50 milliseconds,
will it be able to effectively avoid problems in all environments?  If these are not actual
worries, then I can be quiet.

                  
> Disable Stale Check - Distributed Search (Performance)
> ------------------------------------------------------
>
>                 Key: SOLR-4509
>                 URL: https://issues.apache.org/jira/browse/SOLR-4509
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>         Environment: 5 node SmartOS cluster (all nodes living in same global zone - i.e.
same physical machine)
>            Reporter: Ryan Zezeski
>            Priority: Minor
>         Attachments: SOLR-4509.patch
>
>
> By disabling the Apache HTTP Client stale check I've witnessed a 2-4x increase in throughput
and reduction of over 100ms.  This patch was made in the context of a project I'm leading,
called Yokozuna, which relies on distributed search.
> Here's the patch on Yokozuna: https://github.com/rzezeski/yokozuna/pull/26
> Here's a write-up I did on my findings: http://www.zinascii.com/2013/solr-distributed-search-and-the-stale-check.html
> I'm happy to answer any questions or make changes to the patch to make it acceptable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message