lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <>
Subject [jira] [Updated] (SOLR-1306) Support pluggable persistence/loading of solr.xml details
Date Tue, 30 Oct 2012 16:16:12 GMT


Erick Erickson updated SOLR-1306:

    Attachment: SOLR-1306.patch

MUST be applied _after_ SOLR-1028

OK, here's a preliminary cut at this, no tests yet, but I was looking at logging and it seems
to be doing what I want, putting up for inspection by the curious...

A couple of notes:
1> It turns out that to make this work I needed to incorporate SOLR-4013 and SOLR-3980.
I'd appreciate anyone looking at the synchronization I did around the member variable "loadingCores".
The intent here is to keep two threads from creating the same core at the same time.
1a> I'm assuming that there is exactly one CoreContainer per JVM. Otherwise I don't understand
how any of the synchronization works on the member vars....
1b> Running a simple test things went all to hell in JMX stuff without the synchronization,
apparently the multiple thread problem shows up early and often.
1c> synchronization is always "interesting", so the more eyes the better.
1d> In particular, any good suggestions about bailing out of the sleep loop? Since cores
can take quite a while to warm, I'm having a hard time thinking of a good default. I suppose
another attribute where the provider is mentioned. There's no reason a custom provider has
to be present, so requiring a timeout from the provider doesn't seem workable.

2> I've implemented a trivial CoreDescriptorProvider for a PoC, it's at the bottom. It
pre-supposes you have 4 collections, the accompanying Solr.xml is also below.

3> I'm going to put this away for a couple of hours and come back to it with fresh eyes,
this copy is purely for the curious/critical...

*****sample custom descriptor provider
public class TestCoreContainerProvider implements CoreDescriptorProvider
  public CoreDescriptor getDescriptor(CoreContainer container, String name) {
    if (! "collection2".equals(name) && ! "collection3".equals(name) && !
"collection4".equals(name)) return null;

    CoreDescriptor cd = new CoreDescriptor(container, name, name); // True hack because I
know the dirs are the same as the collection.
    return cd;

  public boolean isPersist(String s) {
    return false;

  public Collection<String> getCoreNames() {
    return new ArrayList<String>(Arrays.asList("collection2", "collection3", "collection4"));

**** solr.xml. Note no ZK stuff.
<solr persistent="false" sharedLib="../../../../../blahblah/out/artifacts/provider_jar">

  <cores adminPath="/admin/cores" defaultCoreName="collection1" host="${host:}" hostPort="${jetty.port:}"
hostContext="${hostContext:}" zkClientTimeout="${zkClientTimeout:15000}" coreDescriptorProviderClass="blah.TestCoreContainerProvider">
    <core name="collection1" instanceDir="collection1" />
> Support pluggable persistence/loading of solr.xml details
> ---------------------------------------------------------
>                 Key: SOLR-1306
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>          Components: multicore
>            Reporter: Noble Paul
>            Assignee: Erick Erickson
>             Fix For: 4.1
>         Attachments: SOLR-1306.patch
> Persisting and loading details from one xml is fine if the no:of cores are small and
the no:of cores are few/fixed . If there are 10's of thousands of cores in a single box adding
a new core (with persistent=true) becomes very expensive because every core creation has to
write this huge xml. 
> Moreover , there is a good chance that the file gets corrupted and all the cores become
unusable . In that case I would prefer it to be stored in a centralized DB which is backed
up/replicated and all the information is available in a centralized location. 
> We may need to refactor CoreContainer to have a pluggable implementation which can load/persist
the details . The default implementation should write/read from/to solr.xml . And the class
should be pluggable as follows in solr.xml
> {code:xml}
> <solr>
>   <dataProvider class="" attr1="val1" attr2="val2"/>
> </solr>
> {code}
> There will be a new interface (or abstract class ) called SolrDataProvider which this
class must implement

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message