lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrien Grand (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
Date Tue, 18 Jun 2019 09:01:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866364#comment-16866364
] 

Adrien Grand commented on LUCENE-8862:
--------------------------------------

I'd like to avoid adding new methods to the collector API for this. I don't think the collector
API needs a visitor API since there aren't many use-cases for collecting multiple collectors
at once via a MultiCollector, especially since Lucene 8 and top-hits retrieval optimizations:
if you want to retrieve both top hits and facets, it is likely a better idea now to run the
query twice concurrently, once with a top-hits collector and another time with a faceting
collector. This will enable the UI to start displaying top hits while facets are still being
computed. Furthermore exposing internals of collectors via a visitor API might prevent some
optimizations or the contract would be a bit weird. For instance MultiLeafCollector removes
its sub collectors when they throw a CollectionTerminatedException, should we visit these
sub collectors or not? We have a similar problem with the Scorer#getChildren API and the BooleanScorer
optimization, which makes the Scorer#getChildren API hard to rely on since it is not consistently
implemented.

> Collector Level Dynamic Memory Accounting
> -----------------------------------------
>
>                 Key: LUCENE-8862
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8862
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Atri Sharma
>            Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which tracks dynamic
memory used by Collectors. This shall allow users to get an accountability as to the memory
usage of their Collectors and better plan their resource capacity. This shall also allow us
to add Collector level limits for memory usage, thus allowing users a finer control over their
resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message