lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (Updated) (JIRA)" <>
Subject [jira] [Updated] (LUCENE-3562) Stop storing TermsEnum in CloseableThreadLocal inside Terms instance
Date Thu, 17 Nov 2011 16:42:51 GMT


Michael McCandless updated LUCENE-3562:

    Attachment: LUCENE-3562.patch

New patch; also cuts over MultiPhraseQuery to save the TermStates from weight -> scorer,
and optimizes BlockTree's TermsEnum to reduce cost of init + seekExact only usages.

I think it's ready!
> Stop storing TermsEnum in CloseableThreadLocal inside Terms instance
> --------------------------------------------------------------------
>                 Key: LUCENE-3562
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 4.0
>         Attachments: LUCENE-3562.patch, LUCENE-3562.patch
> We have sugar methods in (docFreq, totalTermFreq, docs,
> docsAndPositions) that use a saved thread-private TermsEnum to do the
> lookups.
> But on apps that send many threads through Lucene, and/or have many
> segments, this can add up to a lot of RAM, especially if the codecs
> impl holds onto stuff.
> Also, Terms has a close method (closes the CloseableThreadLocal) which
> must be called, but we fail to do so in some places.
> These saved enums are the cause of the recent OOME in TestNRTManager
> (TestNRTManager.testNRTManager -seed
> 2aa27e1aec20c4a2:-4a5a5ecf46837d0e:-7c4f651f1f0b75d7 -mult 3
> -nightly).
> Really sharing these enums is a holdover from before Lucene queries
> would share state (ie, save the TermState from the first pass, and use
> it later to pull enums, get docFreq, etc.).  It's not helpful anymore,
> and it can use gobbs of RAM, so I'd like to remove it.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message