Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8365818448 for ; Thu, 12 Nov 2015 00:05:57 +0000 (UTC) Received: (qmail 54273 invoked by uid 500); 12 Nov 2015 00:05:56 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 54208 invoked by uid 500); 12 Nov 2015 00:05:56 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 54198 invoked by uid 99); 12 Nov 2015 00:05:56 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2015 00:05:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B901E1A5AEF for ; Thu, 12 Nov 2015 00:05:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.991 X-Spam-Level: *** X-Spam-Status: No, score=3.991 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id hckW3gzdCtjY for ; Thu, 12 Nov 2015 00:05:45 +0000 (UTC) Received: from wunderwood.org (wunderwood.org [192.220.101.25]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTP id 2604C439F2 for ; Thu, 12 Nov 2015 00:05:45 +0000 (UTC) Received: (qmail 73082 invoked by uid 25881); 12 Nov 2015 00:05:37 -0000 Received: from unknown (HELO scml-wunder.chegg.com) (wunder@[199.58.143.63]) (envelope-sender ) by 192.220.101.25 (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 12 Nov 2015 00:05:37 -0000 From: Walter Underwood Content-Type: multipart/alternative; boundary="Apple-Mail=_74C63CA1-49AF-4AA5-A58B-AFD3265DBE07" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Sharing a class across cores Date: Wed, 11 Nov 2015 16:05:36 -0800 References: <5643D526.1010200@elyograg.org> To: dev@lucene.apache.org In-Reply-To: X-Mailer: Apple Mail (2.2104) --Apple-Mail=_74C63CA1-49AF-4AA5-A58B-AFD3265DBE07 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Depending on how fast the access needs to be, you could put that big map = in memcache. wunder Walter Underwood wunder@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 11, 2015, at 4:04 PM, Gus Heck wrote: >=20 > P.S. I posted the original message concurrently with the chat = session's occurance I beleive, certainly before I had read it, so no I = haven't actually tried what you suggest yet. >=20 > On Wed, Nov 11, 2015 at 7:02 PM, Gus Heck > wrote: > Yes asked by a colleague :). The chat session is now in our jira = ticket :).=20 >=20 > However, my take on it is that this seems like a pretty broad brush to = paint with to move *all* our classes up and out of the normal core = loading process. I assume there are good reasons for segregating this = stuff into separate class loaders to begin with. It would also be fairly = burdensom to make a separate jar file to break out this one component... >=20 > I really just want a way to stash the map in a place where other cores = can see it (and thus I can appropriately synchronize things so that the = loading only happens once). I'm asking because it seems like surely this = must be a solved problem... if not, it might be easiest to just solve it = by adding some sort of shared resources facility to CoreContainer? >=20 > -Gus >=20 > On Wed, Nov 11, 2015 at 6:54 PM, Shawn Heisey > wrote: > On 11/11/2015 4:11 PM, Gus Heck wrote: > > I have a case where a component loads up a large CSV file (2.5 = million > > lines) to build a map. This worked ok in a case where we had a = single > > core, but it isn't working so well with 40 cores because each core = loads > > a new copy of the component in a new classloader and I get 40 new > > versions of the same class each holding it's own private static = final > > map (one for each core). Each line is small, but a billion of = anything > > gets kinda heavy. Is this the intended class loading behavior? > > > > Is there some where that one can cause a class to be loaded in a = parent > > classloader above the core so that it's loaded just once? I want to = load > > it in some way that leverages standard solr resource loading, so = that > > I'm not hard coding or setting sysprops just to be able to find it. > > > > This is in a copy of trunk from about a month ago... so 6.x stuff is > > mostly available. >=20 > This sounds like a question that I just recently answered on IRC. >=20 > If you remove all elements from your solrconfig.xml files and > place all extra jars for Solr into ${solr.solr.home}/lib ... Solr will > load those jars before any cores are created and they will be = available > to all cores. >=20 > There is a minor bug with this that will be fixed in Solr 5.4.0. It = is > unlikely that this will affect third-party components, but be aware = that > until 5.4, jars in that lib directory will be loaded twice by older = 5.x > versions. >=20 > https://issues.apache.org/jira/browse/SOLR-6188 = >=20 > Thanks, > Shawn >=20 >=20 > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org = > For additional commands, e-mail: dev-help@lucene.apache.org = >=20 >=20 >=20 >=20 > --=20 > http://www.the111shift.com >=20 >=20 > --=20 > http://www.the111shift.com --Apple-Mail=_74C63CA1-49AF-4AA5-A58B-AFD3265DBE07 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Depending on how fast the access needs to be, you could put = that big map in memcache.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


On Nov 11, 2015, at 4:04 PM, Gus Heck <gus.heck@gmail.com> = wrote:

P.S. I posted the original message concurrently = with the chat session's occurance I beleive, certainly before I had read = it, so no I haven't actually tried what you suggest yet.

On Wed, Nov 11, 2015 at 7:02 PM, Gus Heck <gus.heck@gmail.com> wrote:
Yes asked by a colleague :). The chat session = is now in our jira ticket :).

However, my = take on it is that this seems like a pretty broad brush to paint with to = move *all* our classes up and out of the normal core loading process. I = assume there are good reasons for segregating this stuff into separate = class loaders to begin with. It would also be fairly burdensom to make a = separate jar file to break out this one component...

I really just want a way to stash the map in a place where = other cores can see it (and thus I can appropriately synchronize things = so that the loading only happens once). I'm asking because it seems like = surely this must be a solved problem... if not, it might be easiest to = just solve it by adding some sort of shared resources facility to = CoreContainer?

-Gus

On Wed, Nov 11, = 2015 at 6:54 PM, Shawn Heisey <apache@elyograg.org> wrote:
On = 11/11/2015 4:11 PM, Gus Heck wrote:
> I have a case where a component loads up a large CSV file (2.5 = million
> lines) to build a map. This worked ok in a case where we had a = single
> core, but it isn't working so well with 40 cores because each core = loads
> a new copy of the component in a new classloader and I get 40 = new
> versions of the same class each holding it's own private static = final
> map (one for each core). Each line is small, but a billion of = anything
> gets kinda heavy. Is this the intended class loading behavior?
>
> Is there some where that one can cause a class to be loaded in a = parent
> classloader above the core so that it's loaded just once? I want to = load
> it in some way that leverages standard solr resource loading, so = that
> I'm not hard coding or setting sysprops just to be able to find = it.
>
> This is in a copy of trunk from about a month ago... so 6.x stuff = is
> mostly available.

This sounds like a question that I just recently answered on = IRC.

If you remove all <lib> elements from your solrconfig.xml files = and
place all extra jars for Solr into ${solr.solr.home}/lib ... Solr = will
load those jars before any cores are created and they will be = available
to all cores.

There is a minor bug with this that will be fixed in Solr 5.4.0.  = It is
unlikely that this will affect third-party components, but be aware = that
until 5.4, jars in that lib directory will be loaded twice by older = 5.x
versions.

https://issues.apache.org/jira/browse/SOLR-6188

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org




--
http://www.the111shift.com



--
http://www.the111shift.com

= --Apple-Mail=_74C63CA1-49AF-4AA5-A58B-AFD3265DBE07--