lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Earwin Burrfoot (JIRA)" <>
Subject [jira] Commented: (LUCENE-2355) Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
Date Thu, 01 Apr 2010 15:21:27 GMT


Earwin Burrfoot commented on LUCENE-2355:

Some more stuff I forgot:

- No need to keep around origInstance in CoreReaders. It is used to purge FieldCache, but
the cache is purged based on getFieldCacheKey, which is common across all clones/reopens
- MultiReader-level norms cache is not needed with per-segment searching and is removed. The
method that extracts norms into a user-passed array still has the same performance even without
the cache.

> Refactor Directory/Multi/SegmentReader creation/reopening/cloning/closing
> -------------------------------------------------------------------------
>                 Key: LUCENE-2355
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Earwin Burrfoot
> *Reader lifecycle evolved over time to become some heavily tangled mess. It's hard to
understand what's going on there, it's even harder to add some fields/logic while ensuring
that all possible code paths preserve these fields/interact with the logic properly. While
some of said mess is justified by the task at hand, a big part is just badly done copypaste
and can be removed.
> I am currently refactoring this and intended to open an issue with a working patch, but
the task winded up somewhat bigger than I expected, so I'm opening it earlier to track stuff
> The list is by no means exhaustive.
> - an iteration to create SRs is copypasted several times, one of them (IW) with wrong
iteration bound
> - it is also overly complex and can be folded for create/reopen cases
> - readers sent to IndexReaderWarmer are termindexless/docstoreless on some occasions
> - it is possible to clone() your way to readwrite NRT reader
> - IndexDeletionPolicy is not always preserved through clones/reopens
> - cloned readers share CoreReaders and, consequently, updated termsIndex/docStores
> - threadlocal versions of fieldsReader/termsVector are bound to SR, not CoreReaders and
thus are recreated on clone/reopen
> - double-initialization for some fields (someone got lost and did this to be sure I guess),
stupid assert checks ( qwe = new(); assert qwe != null )
> - SR is not always recreated when compound status of underlying segment changes
> - deleting already deleted doc marks deletions dirty and rewrites them
> - lots of synchronization is done around Reader, while it can be narrowed down to norms/deletions/whatever
> I did some structural modifications:
> - CompositeReader extracts common code from DirectoryReader and MultiReader (complete)
> - ReadonlyDirectoryReader and ReadonlySegmentReader are dead, MutableD/SReaders are introduced
and carry all modification logic/fields (DR complete, SR in progress)
> - WriterBackedReader encapsulates NRT reader logic (complete)
> - CoreReaders split into CoreReaders, DocStores, TermInfos. All of these are immutable
and SR is cloned when you need to change its mode (in progress)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message