lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Smiley <david.w.smi...@gmail.com>
Subject Re: How hard would a "wipe all deletes" operation be?
Date Tue, 16 Aug 2016 16:56:35 GMT
Good points Jeff.

On Tue, Aug 16, 2016 at 12:37 PM Jeff Wartes <jwartes@whitepages.com> wrote:

>
>
> Looks to me like if you’re using TieredMergePolicy,
> forceMergeDeletesPctAllowed is simply a way of including a segment as a
> merge candidate that would not have otherwise been a candidate (due to
> being a different tier). Setting it to zero means never include a segment
> as a merge candidate due to delete pct, setting it to 100 means any segment
> with a delete is a candidate.
>
> Related, TieredMergePolicy .reclaimDeletesWeight alters the likelihood that two candidate
segments will be selected for a merge based on delete ratio.
>
> Related, doing optimize with expungeDeletes=true simply causes
> TieredMergePolicy to use the forceMergeDeletesPctAllowed.
>
>
>
> Nothing here really answers Shawn’s use case though, since in all cases
> you’re still required to merge segments to remove deletes. Clearly, we
> could set a very high forceMergeDeletesPctAllowed and reclaimDeletesWeight,
> but also clearly, if we really wanted to remove all deletes, an optimize
> with maxSegments=1 would do the job. It’s just expensive, and it’ll be that
> expensive again as soon as you get more deletes.
>
>
>
> In my mind, I’d want to say “if deletes are more than x% of a given
> segment’s size, just re-write the segment while filtering out the deletes.
> The only reason I might care about the merge policy is that I might prefer
> to do this only on the larger/longer-lived segments that don’t get merged
> often.
>
>
>
>
>
>
>
> *From: *David Smiley <david.w.smiley@gmail.com>
> *Reply-To: *"dev@lucene.apache.org" <dev@lucene.apache.org>
> *Date: *Monday, August 15, 2016 at 8:56 PM
> *To: *"dev@lucene.apache.org" <dev@lucene.apache.org>
> *Subject: *Re: How hard would a "wipe all deletes" operation be?
>
>
>
> Shawn:
>
>
> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers
> Search for "expungeDeletes".  So it's 10% by default.  I did some digging
> and I see this 10% figure is settable on the TieredMergePolicy.  So you
> could modify solrconfig.xml and
> set <forceMergeDeletesPctAllowed>0</forceMergeDeletesPctAllowed> as a
> setting on the merge policy.
>
>
>
> On Thu, Aug 11, 2016 at 2:59 PM Shawn Heisey <apache@elyograg.org> wrote:
>
> On 8/11/2016 10:58 AM, David Smiley wrote:
> > Note there is a threshold to expungeDeletes such that if there aren't
> > enough deletes in a segment relative to the docs in that segments, it
> > won't do any expunging.
>
> What sort of request do I need to send to force expunging *all* deleted
> documents, even if there's only one in a segment?  Is that possible?
>
> That's the end goal of all this scheming.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
>
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Mime
View raw message