lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Noble Paul (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-8349) Allow sharing of large in memory data structures across cores
Date Sun, 21 Feb 2016 21:27:18 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156236#comment-15156236
] 

Noble Paul edited comment on SOLR-8349 at 2/21/16 9:26 PM:
-----------------------------------------------------------

[~gus_heck] There is another easy existing solution for this problem

How to do this

1) Store your large file in the blob store
2) Use {{blobRef = JarRepository.getJarIncRef(name)}} to get the content (we will change the
method names for it to make sense for you)
3) Make your component register as a closehook
4) In the {{postClose()}}, do a {{blobref.decrementJarRefCount()}}


The advantages of this solution are,

1) You get a version managed store for your large files without screwing up your ZK
2) There are APIs to manage the blob
3) It is already refcounted etc

caveats are

1) It does not work for Standalone. We can extend it to do that
2) You will have to make your component {{startup=lazy}}


was (Author: noble.paul):
[~gus_heck] There is another easy existing solution for this problem

How to do this

1) Store your large file in the blob store
2) Use {{blobRef = JarRepository.getJarIncRef(name)}} to get the content (we will change the
method names for it to make sense for you)
3) Make your component register as a closehook
4) In the {{postClose()}}, do a {{blobref.decrementJarRefCount()}}


The advantages of this solution are,

1) You get a version managed store for your large files without screwing up your ZK
2) It is already refcounted etc

caveats are

1) It does not work for Standalone. We can extend it to do that
2) You will have to make your component {{startup=lazy}}

> Allow sharing of large in memory data structures across cores
> -------------------------------------------------------------
>
>                 Key: SOLR-8349
>                 URL: https://issues.apache.org/jira/browse/SOLR-8349
>             Project: Solr
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 5.3
>            Reporter: Gus Heck
>         Attachments: SOLR-8349.patch, SOLR-8349.patch
>
>
> In some cases search components or analysis classes may utilize a large dictionary or
other in-memory structure. When multiple cores are loaded with identical configurations utilizing
this large in memory structure, each core holds it's own copy in memory. This has been noted
in the past and a specific case reported in SOLR-3443. This patch provides a generalized capability,
and if accepted, this capability will then be used to fix SOLR-3443.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message