Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of solrexpert@gmail.com
 designates 209.85.217.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAB_8Yd-HxdiyEwz4jcqCf1b2h8ajOxv36A27YbznYh8EL2HEug@mail.gmail.com>
References: 
 <CANutaR-XNvObA0_M+7dKY8qxx8UrRKSrfauXxhJ8ekJOj5EZyQ@mail.gmail.com>
	<CAB_8Yd-HxdiyEwz4jcqCf1b2h8ajOxv36A27YbznYh8EL2HEug@mail.gmail.com>
Date: Mon, 3 Nov 2014 19:27:17 +0200
Message-ID: 
 <CAHUAEU_qDvpz2gTjABcRZKr-VfKab6zXXc6Hn9YfTR_qCaQh3Q@mail.gmail.com>
Subject: Re: Solr slow start up (tlog is small)
From: Dmitry Kan <solrexpert@gmail.com>
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Content-Type: multipart/alternative; boundary=001a1134716889bafb0506f7a829

--001a1134716889bafb0506f7a829
Content-Type: text/plain; charset=UTF-8

One other reason for a slow start-up can be large number of segments in the
index. Which I'm guessing is not the case since you optimized? But anyway,
what's the number of segments in both 280G and 185G indices?

Dmitry

On Mon, Nov 3, 2014 at 6:17 PM, Yonik Seeley <yonik@heliosearch.com> wrote:

> Can you tell from the logs what Solr is doing during that time?
> Do you have any warming queries configured?
> Also see this: https://issues.apache.org/jira/browse/SOLR-6679
>   (comment out suggester related stuff if you aren't using it)
>
> -Yonik
> http://heliosearch.org - native code faceting, facet functions,
> sub-facets, off-heap data
>
>
> On Mon, Nov 3, 2014 at 11:03 AM, Po-Yu Chuang <ratbert.chuang@gmail.com>
> wrote:
> > Hi,
> >
> > I am using Solr 4.9 with Tomcat and it works fine except that the
> > deployment of solr.war is too long. While deploying Solr, all webapps on
> > Tomcat stop responding which is unacceptable. Most articles I found say
> > that it might result from big transaction log because of uncommitted
> > documents, but this is not my case.
> >
> > At first, the Solr data is 280G and the start up time is 30 minutes.
> Then I
> > set a field to stored="false" and re-index whole data. The data size
> became
> > 185G and the start up time reduced to 17 minutes, but it is still too
> long.
> >
> > Here are some numbers I measured:
> >
> > 1)
> > Solr home: 280G
> > tlog: 500K
> > 30 min to start up
> > While starting up, disk read is constantly about 50MB/s (according to
> > dstat). So it seems that Solr reads 30m * 60s * 50MB/s = 90GB of data
> while
> > starting up, which is 30% of index data size.
> >
> > 2)
> > Solr home: 185G
> > tlog: 5M
> > 17 minutes to start up
> > While starting up, disk read is constantly about 5MB/s (according to
> > dstat). So it seems that Solr reads 17m * 60s *5MB/s = 5GB of data while
> > starting up, which is about 3% of index data size.
> >
> > p.s. I did commit each time 1000 documents being added and did
> optimization
> > after all documents are added.
> >
> > Any ideas or suggestions would be appreciated.
> >
> > Thanks,
> > Po-Yu
>


-- 
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info

--001a1134716889bafb0506f7a829--