lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gus Heck (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores
Date Mon, 30 Nov 2015 16:05:11 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15032007#comment-15032007
] 

Gus Heck commented on SOLR-8349:
--------------------------------

Further background for this feature... a precursor version of this patch (which does not have
the interface and thus can't fix SOLR-3443) is in use at a client where we had a ~900MB hash
map for looking up lat/lon from custom query parameters. This map was needed for all our cores.
This is anticipated to save >35GB of ram since 40+ cores will live on a machine. The original
implementation of this lat/lon lookup feature for the client attempted to use a static field,
but the independent class loaders (each core gets it's own class loader) loaded fresh copies
of the class each with it's own static map.

It's worth noting that the analyzers such as the hunspell one in SOLR-3443 are not loaded
by the core's class loaders and the excess memory there is held in a member field per instance,
so a static variable based solution would be possible there. I thought it was better to provide
a uniform solution. 

Another possible follow on feature (or perhaps enhancement to this one) would be a means of
reference counting the shared resources and removing them. In the present (initial) patch,
a long running solr instance where lots of cores are added and removed would potentially have
unused container resources hanging around (though they would become used again with no loading
time if a core were re-installed that required them). I didn't go into the complexity of removal
because I wasn't sure if it would be deemed necessary.


> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
>                 Key: SOLR-8349
>                 URL: https://issues.apache.org/jira/browse/SOLR-8349
>             Project: Solr
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 5.3
>            Reporter: Gus Heck
>         Attachments: SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large dictionary or
other in-memory structure. When multiple cores are loaded with identical configurations utilizing
this large in memory structure, each core holds it's own copy in memory. This has been noted
in the past and a specific case reported in SOLR-3443. This patch provides a generalized capability,
and if accepted, this capability will then be used to fix SOLR-3443.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message