lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <wun...@wunderwood.org>
Subject Re: Sharing a class across cores
Date Thu, 12 Nov 2015 00:05:36 GMT
Depending on how fast the access needs to be, you could put that big map in memcache.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 11, 2015, at 4:04 PM, Gus Heck <gus.heck@gmail.com> wrote:
> 
> P.S. I posted the original message concurrently with the chat session's occurance I beleive,
certainly before I had read it, so no I haven't actually tried what you suggest yet.
> 
> On Wed, Nov 11, 2015 at 7:02 PM, Gus Heck <gus.heck@gmail.com <mailto:gus.heck@gmail.com>>
wrote:
> Yes asked by a colleague :). The chat session is now in our jira ticket :). 
> 
> However, my take on it is that this seems like a pretty broad brush to paint with to
move *all* our classes up and out of the normal core loading process. I assume there are good
reasons for segregating this stuff into separate class loaders to begin with. It would also
be fairly burdensom to make a separate jar file to break out this one component...
> 
> I really just want a way to stash the map in a place where other cores can see it (and
thus I can appropriately synchronize things so that the loading only happens once). I'm asking
because it seems like surely this must be a solved problem... if not, it might be easiest
to just solve it by adding some sort of shared resources facility to CoreContainer?
> 
> -Gus
> 
> On Wed, Nov 11, 2015 at 6:54 PM, Shawn Heisey <apache@elyograg.org <mailto:apache@elyograg.org>>
wrote:
> On 11/11/2015 4:11 PM, Gus Heck wrote:
> > I have a case where a component loads up a large CSV file (2.5 million
> > lines) to build a map. This worked ok in a case where we had a single
> > core, but it isn't working so well with 40 cores because each core loads
> > a new copy of the component in a new classloader and I get 40 new
> > versions of the same class each holding it's own private static final
> > map (one for each core). Each line is small, but a billion of anything
> > gets kinda heavy. Is this the intended class loading behavior?
> >
> > Is there some where that one can cause a class to be loaded in a parent
> > classloader above the core so that it's loaded just once? I want to load
> > it in some way that leverages standard solr resource loading, so that
> > I'm not hard coding or setting sysprops just to be able to find it.
> >
> > This is in a copy of trunk from about a month ago... so 6.x stuff is
> > mostly available.
> 
> This sounds like a question that I just recently answered on IRC.
> 
> If you remove all <lib> elements from your solrconfig.xml files and
> place all extra jars for Solr into ${solr.solr.home}/lib ... Solr will
> load those jars before any cores are created and they will be available
> to all cores.
> 
> There is a minor bug with this that will be fixed in Solr 5.4.0.  It is
> unlikely that this will affect third-party components, but be aware that
> until 5.4, jars in that lib directory will be loaded twice by older 5.x
> versions.
> 
> https://issues.apache.org/jira/browse/SOLR-6188 <https://issues.apache.org/jira/browse/SOLR-6188>
> 
> Thanks,
> Shawn
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org <mailto:dev-unsubscribe@lucene.apache.org>
> For additional commands, e-mail: dev-help@lucene.apache.org <mailto:dev-help@lucene.apache.org>
> 
> 
> 
> 
> -- 
> http://www.the111shift.com <http://www.the111shift.com/>
> 
> 
> -- 
> http://www.the111shift.com <http://www.the111shift.com/>

Mime
View raw message