lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: throttle segment merging
Date Sun, 28 Oct 2012 23:09:02 GMT
1) Do you use compound files (CFS)? This adds a lot of overhead to merging.
2) Does ES use the same merge policy code as Solr?

In solrconfig.xml, here are the lines that control segment merging. You can probably set mergeFactor
to 20 and cut the amount of disk I/O.

    <!-- Expert: Merge Policy 
         The Merge Policy in Lucene controls how merging of segments is done.
         The default since Solr/Lucene 3.3 is TieredMergePolicy.
         The default since Lucene 2.3 was the LogByteSizeMergePolicy,
         Even older versions of Lucene used LogDocMergePolicy.
      -->
    <!--
        <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
          <int name="maxMergeAtOnce">10</int>
          <int name="segmentsPerTier">10</int>
        </mergePolicy>
      -->
       
    <!-- Merge Factor
         The merge factor controls how many segments will get merged at a time.
         For TieredMergePolicy, mergeFactor is a convenience parameter which
         will set both MaxMergeAtOnce and SegmentsPerTier at once.
         For LogByteSizeMergePolicy, mergeFactor decides how many new segments
         will be allowed before they are merged into one.
         Default is 10 for both merge policies.
      -->
    <!-- 
    <mergeFactor>10</mergeFactor>
      -->

    <!-- Expert: Merge Scheduler
         The Merge Scheduler in Lucene controls how merges are
         performed.  The ConcurrentMergeScheduler (Lucene 2.3 default)
         can perform merges in the background using separate threads.
         The SerialMergeScheduler (Lucene 2.2 default) does not.
     -->
    <!-- 
       <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
       -->


----- Original Message -----
| From: "Radim Kolar" <hsn@filez.com>
| To: solr-user@lucene.apache.org
| Sent: Saturday, October 27, 2012 7:44:46 PM
| Subject: Re: throttle segment merging
| 
| Dne 26.10.2012 3:47, Tomás Fernández Löbbe napsal(a):
| >> Is there way to set-up logging to output something when segment
| >> merging
| >> runs?
| >>
| > I think segment merging is logged when you enable infoStream
| > logging (you
| > should see it commented in the solrconfig.xml)
| no, segment merging is not logged at info level. it needs customized
| log
| config.
| 
| >
| >> Can be segment merges throttled?
|  > You can change when and how segments are merged with the merge
| policy, maybe it's enough for you changing the initial settings
| (mergeFactor for example)?
| 
| I am now researching elasticsearch, it can do it, its lucene 3.6
| based
| 

Mime
View raw message