lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Karich <peat...@yahoo.de>
Subject Re: Solr Indexing slows down
Date Mon, 02 Aug 2010 10:19:12 GMT
Thanks Otis, for this clarification!

That means I will have to descrease the commit frequency to speed up
indexing.
How could I do this if I don't want to introduce an artificial delay
time? ... via increasing the batch size?

Today I have read in another thread[1] that one should univert the
field? What is it and how can I do this?

Regards,
Peter.

[1]
http://www.mail-archive.com/solr-user@lucene.apache.org/msg36113.html


> As you make changes to your index, you probably want to see the new/modified 
> documents in your search results.  In order to do that, the new searcher needs 
> to be reopened, and this happens on commit.
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
>   
>> From: Peter Karich <peathal@yahoo.de>
>> To: solr-user@lucene.apache.org
>> Sent: Fri, July 30, 2010 6:19:03 PM
>> Subject: Re: Solr Indexing slows down
>>
>> Hi Otis,
>>
>> does it mean that a new searcher is opened after I commit?
>> I  thought only on startup...(?)
>>
>> Regards,
>> Peter.
>>
>>     
>>> Peter, there  are events in solrconfig where you define warm up queries when

>>>       
>> a 
>>
>>     
>>> new  searcher is opened.
>>>
>>> There are also cache settings that play a  role here.
>>>
>>> 30-60 seconds is pretty frequent for  Solr.
>>>
>>> Otis
>>> ----
>>> Sematext :: http://sematext.com/ :: Solr -  Lucene - Nutch
>>> Lucene ecosystem search :: http://search-lucene.com/
>>>
>>>
>>>
>>> ----- Original  Message ----
>>>  
>>>       
>>>> From: Peter Karich <peathal@yahoo.de>
>>>> To: solr-user@lucene.apache.org
>>>>  Sent: Fri, July 30, 2010 4:06:48 PM
>>>> Subject: Re: Solr Indexing  slows down
>>>>
>>>> Hi Erick!
>>>>
>>>> thanks for  the response!
>>>> I will answer your questions   ;-)
>>>>
>>>>    
>>>>         
>>>>> How often are you  making changes to your index?
>>>>>      
>>>>>           
>>>>  Every  30-60 seconds. Too heavy?
>>>>
>>>>
>>>>     
>>>>         
>>>>> Do you have autocommit  on?
>>>>>       
>>>>>           
>>>> No.
>>>>
>>>>
>>>>    
>>>>         
>>>>> Do you commit when updating each   document?
>>>>>      
>>>>>           
>>>> No. I commit after a  batch update of 200  documents
>>>>
>>>>
>>>>     
>>>>         
>>>>> Committing too often and consequently firing off   warmup queries is
the 
>>>>>           
>> first 
>>
>>     
>>>>>      
>>>>>           
>>>>  place I'd look.
>>>>
>>>> Why is commiting firing  warmup  queries? Is there any documentation about
>>>> this subject?
>>>>  How can I  be sure that the previous commit has done its  magic?
>>>>
>>>>    
>>>>         
>>>>> there are   several config values that influence the commit frequency
>>>>>       
>>>>>           
>>>> I now know  the autowarm and the  mergeFactor config. What else? Is this
>>>> documentation   complete:
>>>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?
>>>>
>>>> Regards,
>>>>  Peter.
>>>>
>>>>    
>>>>         
>>>>> See the subject  about 1500 threads. The  first place I'd look is how
>>>>> often  you're committing. If you're  committing before the warmup  queries
>>>>> from the previous commit have done  their magic,  you might be getting
>>>>> into a death  spiral.
>>>>>
>>>>>  HTH
>>>>>  Erick
>>>>>
>>>>> On Thu, Jul 29, 2010 at 7:02 AM, Peter  Karich  <peathal@yahoo.de>
  
>>>>>           
> wrote:
>   
>>>>>  
>>>>>       
>>>>>           
>>>>>> Hi,
>>>>>>
>>>>>> I  am  indexing a solr 1.4.0 core and commiting gets slower and 
slower.
>>>>>>  Starting from 3-5 seconds for ~200 documents  and ending with over
60
>>>>>>  seconds after 800 commits.  Then, if I reloaded the index, it is
as  
>>>>>>             
> fast
>   
>>>>>> as  before! And today I have read a similar thread [1] and  indeed:
if  I
>>>>>> set autowarming for the caches to 0 the slowdown   disappears.
>>>>>>
>>>>>> BUT at the same time I would  like to offer  searching on that core,

>>>>>>             
> which
>   
>>>>>> would be  dramatically slowed down (due  to no  autowarming).
>>>>>>
>>>>>> Does someone know a better  solution  to avoid index-slow-down?
>>>>>>
>>>>>>  Regards,
>>>>>>   Peter.
>>>>>>
>>>>>> [1] http://www.mail-archive.com/solr-user@lucene.apache.org/msg20785.html
>>>>>>
>>>>>>            
>>>>>>             


Mime
View raw message