lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martijn van Groningen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-2725) TieredMergePolicy and expungeDeletes behaviour
Date Sat, 20 Aug 2011 14:31:27 GMT

     [ https://issues.apache.org/jira/browse/SOLR-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Martijn van Groningen updated SOLR-2725:
----------------------------------------

    Description: 
During executing a commit with expungeDeletes I noticed there were still a lot of segments
left.
However there were still ~30 segments left with deletes after the commit finished.

After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed
isn't invoked.
I think the following statements in SolrIndexConfig#buildMergePolicy method will purge all
deletes:
{code}
tieredMergePolicy.setExpungeDeletesPctAllowed(0);
{code} 
This also reflects the behavior of Solr 3.1 / 3.2

After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best for
performance:
http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120

I think we should add an option to solrconfig.xml that allows users to set this option to
whatever value is best for them:
{code:xml}
<expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
{code}

Also having a expungeDeletesPctAllowed per commit command would be great:
{code:xml}
<commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
{code}

  was:
During executing a commit with expungeDeletes I noticed there were still a lot of segments
left.
My solconfig.xml (that uses TieredMergePolicy, b/c luceneMatchVersion=3.3) contained:
{code:xml}
<indexDefaults>
   <mergeFactor>5</mergeFactor>
{code}
However there were still ~30 segments left with deletes after the commit finished.

After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setMaxMergeAtOnceExplicit
and
TieredMergePolicy#setExpungeDeletesPctAllowed aren't invoked.
I think the following statements should be added to the SolrIndexConfig#buildMergePolicy method:
{code}
tieredMergePolicy.setMaxMergeAtOnceExplicit(mergeFactor);
tieredMergePolicy.setExpungeDeletesPctAllowed(0);
{code} 
I think these changes should reflect the behavior of Solr 3.1 / 3.2


> TieredMergePolicy and expungeDeletes behaviour
> ----------------------------------------------
>
>                 Key: SOLR-2725
>                 URL: https://issues.apache.org/jira/browse/SOLR-2725
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 3.3
>            Reporter: Martijn van Groningen
>             Fix For: 3.4, 4.0
>
>
> During executing a commit with expungeDeletes I noticed there were still a lot of segments
left.
> However there were still ~30 segments left with deletes after the commit finished.
> After looking in SolrIndexConfig class I noticed that TieredMergePolicy#setExpungeDeletesPctAllowed
isn't invoked.
> I think the following statements in SolrIndexConfig#buildMergePolicy method will purge
all deletes:
> {code}
> tieredMergePolicy.setExpungeDeletesPctAllowed(0);
> {code} 
> This also reflects the behavior of Solr 3.1 / 3.2
> After some discussion on IRC setting expungeDeletesPctAllowed always to zero isn't best
for performance:
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-08-20#l120
> I think we should add an option to solrconfig.xml that allows users to set this option
to whatever value is best for them:
> {code:xml}
> <expungeDeletesPctAllowed>0</expungeDeletesPctAllowed>
> {code}
> Also having a expungeDeletesPctAllowed per commit command would be great:
> {code:xml}
> <commit waitFlush="false" waitSearcher="false" expungeDeletes="true" expungeDeletesPctAllowed="0"/>
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message