lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chantal Ackermann <chantal.ackerm...@btelligent.de>
Subject mergeFactor / indexing speed
Date Fri, 31 Jul 2009 12:04:03 GMT
Dear all,

I want to find out which settings give the best full index performance 
for my setup.
Therefore, I have been running a small index (less than 20k documents) 
with a mergeFactor of 10 and 100.
In both cases, indexing took about 11.5 min:

mergeFactor: 10
<str name="Time taken ">0:11:46.792</str>
mergeFactor: 100
/admin/cores?action=RELOAD
<str name="Time taken ">0:11:44.441</str>
Tomcat restart
<str name="Time taken ">0:11:34.143</str>

This is a Tomcat 5.5.20, started with a max heap size of 1GB. But it 
always used much less. No swapping (RedHat Linux 32bit, 3GB RAM, old ATA 
disk).


Now, I have three questions:

1. How can I check which mergeFactor is really being used? The 
solrconfig.xml that is displayed in the admin application is the 
up-to-date view on the file system. I tested that. But it's not 
necessarily what the current SOLR core is using, isn't it?
Is there a way to check on the actually used mergeFactor (while the 
index is running)?
2. I changed the mergeFactor in both available settings (default and 
main index) in the solrconfig.xml file of the core I am reindexing. That 
is the correct place? Should a change in performance be noticeable when 
increasing from 10 to 100? Or is the change not perceivable if the 
requests for data are taking far longer than all the indexing itself?
3. Do I have to increase rumBufferSizeMB if I increase mergeFactor? (Or 
some other setting?)

(I am still trying to get profiling information on how much application 
time is eaten up by db connection/requests/processing.
The root entity query is about (average) 20ms. The child entity query is 
less than 10ms.
I have my custom entity processor running on the child entity that 
populates the map using a multi-row result set. I have also attached one 
regex and one script transformer.)

Thank you for any tips!
Chantal



-- 
Chantal Ackermann

Mime
View raw message