lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-7834) BlockTree's terms index should be loaded into memory lazily
Date Thu, 01 Jun 2017 15:08:04 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033127#comment-16033127
] 

David Smiley commented on LUCENE-7834:
--------------------------------------

Making something _not possible_ seems like an extreme position to take, especially considering
that search isn't a one-size (one decision) fits-all technology, and Lucene is a library not
a search server.  That's one aspect of search I find so interesting and fun; each user often
has some unique aspects that make some defaults less appropriate for them.  Perhaps a search
app might want to work in _very_ low memory (relative to the data) environments (or very big
data relative to realistic RAM amounts on a machine), and they wish to spend their resources
engineering it to work and not buying more RAM.  Or perhaps they want even lower NRT search
tolerances in which the term dictionaries are eagerly loaded in a non-blocking thread.  I
understand it's possible to shoot one's self in the foot, so to speak, but I think that's
more a question of defaults vs letting users who know and test their environment be able to
do what they want to do without forking large parts of Lucene.

> BlockTree's terms index should be loaded into memory lazily
> -----------------------------------------------------------
>
>                 Key: LUCENE-7834
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7834
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/codecs
>            Reporter: David Smiley
>            Assignee: David Smiley
>         Attachments: LUCENE_7834_BlockTreeLazyFST.patch
>
>
> I propose that BlockTree delay loading the FST prefix terms index into memory until {{terms(String)}}
is first called.  This seems like how it should work since if someone wants to eager load
then they can use {{IndexReaderWarmer}}.  By making the FST lazy load, we can be more NRT
friendly (if some fields are rarely used), and also save memory (if some fields are rarely
used).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message