Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of johannes.goll@gmail.com
 designates 209.85.218.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=H01A500lhRjEB+WYYGImL1SDPhr+rmus+Vc9b5NLCew9+diFP7/vz2HKyr/bnXeNy8
         3cAEXzK5TpMVVNTKsq6onA3HzkfyBVGzcyWgTdjBhl7j6tXCUNuoHmwjtSbsXacMnYZr
         OuSrEng7MGTELBm0CYzjbO9Jp78VLtOSiOOaA=
MIME-Version: 1.0
In-Reply-To: <BANLkTi=cvjVdoYUT8C_p=HVvysX7jELagQ@mail.gmail.com>
References: <58B6CA2F3B4F49D2A657CC204D24C2E0@gmail.com>
	<5161313A-16D5-4C44-9C0E-9FDC2DF7852C@apache.org>
	<5E130813C7C245FA8EC6940DE003E64E@gmail.com>
	<67A5508302F642F483E2D086379FDEBB@gmail.com>
	<1306411887962-2988464.post@n3.nabble.com>
	<BANLkTi=7eY2R6YFpWauN-rDH+fq0VCTTYQ@mail.gmail.com>
	<BANLkTi=cvjVdoYUT8C_p=HVvysX7jELagQ@mail.gmail.com>
Date: Tue, 14 Jun 2011 09:44:55 -0400
Message-ID: <BANLkTi=rLJqy8uh973Paw=3AFN6Acn-YpQ@mail.gmail.com>
Subject: Re: Huge performance drop in distributed search w/ shards on the same
 server/container
From: Johannes Goll <johannes.goll@gmail.com>
To: solr-user@lucene.apache.org, yonik@lucidimagination.com
Content-Type: multipart/alternative; boundary=20cf305e24edc1814304a5ac3de5

--20cf305e24edc1814304a5ac3de5
Content-Type: text/plain; charset=ISO-8859-1

I increased the maximum POST size and headerBufferSize to 10MB ; lowThreads
to 50, maxThreads to 100000 and lowResourceMaxIdleTime=15000. We tried
tomcat 6 using the following Connnector settings :

<Connector port="8989"
       protocol="HTTP/1.1"
       redirectPort="8443"
       URIEncoding="UTF-8"
       maxThreads="10000"
       maxKeepAliveRequests="200"
       maxHttpHeaderSize="10485760"
       />

I am getting the same exception as for jetty

SEVERE: org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: java.net.SocketException:
Connection reset

This seem to point towards a Solr specific issue (solrj.SolrServerException
during individual shard searches).  I monitored the CPU utilization
executing sequential distributed searches and noticed that in the beginning
all CPUs are getting used for a short period of time (multiple lines for
shard searches are shown in the log with isShard=true arguments), then all
CPU except one become idle and the request is being processed by this one
CPU for the longest period of time.

I also noticed in the logs that while most of the individual shard searches
(isShard=true) have low QTimes (5-10), a minority has extreme QTimes
(104402-105126). All shards are fairly similar in size and content (1.2 M
documents) and the StatsComponent is being used
[stats=true&stats.field=weight&stats.facet=library_id]. Here library_id
equals the shard/core name.

Is there an internal timeout for gathering shard results or other fixed
resource limitation ?

Johannes


2011/6/13 Yonik Seeley <yonik@lucidimagination.com>

> On Sun, Jun 12, 2011 at 9:10 PM, Johannes Goll <johannes.goll@gmail.com>
> wrote:
> > However, sporadically, Jetty 6.1.2X (shipped with  Solr 3.1.)
> > sporadically throws Socket connect exceptions when executing distributed
> > searches.
>
> Are you using the exact jetty.xml that shipped with the solr example
> server,
> or did you make any modifications?
>
> -Yonik
> http://www.lucidimagination.com
>


-- 
Johannes Goll
211 Curry Ford Lane
Gaithersburg, Maryland 20878

--20cf305e24edc1814304a5ac3de5--