lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-2655) Get deletes working in the realtime branch
Date Sat, 02 Oct 2010 09:46:34 GMT


Michael McCandless commented on LUCENE-2655:

bq. Ok, I have been stuck/excited about not having to use/understand the remap-docids method,
because it's hard to debug. However I see what you're saying, and why remap-docids exists.
I'll push the DWP buffered deletes to the flushed deletes.

I think we still must remap, at least on the pushed (deletesFlushed) deletes?

On the buffered deletes for the DWPT (deletesInRAM), I think we can make these relative to
the DWPT (ie start from 0), but on pushing them into flushed deletes we re-base them?

bq. This large cost is from loading the terms index and deleted docs?

Yes.  We don't (hopefully) load norms, field cache, etc.

bq. When those large segments are merged though, the IO cost is so substantial that loading
tii or del into RAM probably doesn't account for much of the aggregate IO, they're probably
in the noise?

Well, the applyDeletes method is sync'd, vs merging which is fully concurrent.  (Also, merging
doesn't load the tii).

bq. Or are you referring to the NRT apply deletes flush, however that is on a presumably pooled

Right, it would be pooled for the NRT case, so this is only a (sizable) perf problem for the
non-nrt case.

bq. Or you're just saying that today we're applying deletes across the board to all segments
prior to a merge, regardless of whether or not they're even involved in the merge? It seems
like that is changeable?

Right!  That's what we do today (apply deletes to all segs) whereas it's really only necessary
to apply them to the segments being merged.  I opened LUCENE-2680 to track this.

> Get deletes working in the realtime branch
> ------------------------------------------
>                 Key: LUCENE-2655
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: Realtime Branch
>            Reporter: Jason Rutherglen
>             Fix For: Realtime Branch
>         Attachments: LUCENE-2655.patch
> Deletes don't work anymore, a patch here will fix this.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message