lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-1306) Support pluggable persistence/loading of solr.xml details
Date Tue, 30 Oct 2012 16:16:12 GMT

     [ https://issues.apache.org/jira/browse/SOLR-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Erick Erickson updated SOLR-1306:
---------------------------------

    Attachment: SOLR-1306.patch

MUST be applied _after_ SOLR-1028

OK, here's a preliminary cut at this, no tests yet, but I was looking at logging and it seems
to be doing what I want, putting up for inspection by the curious...

A couple of notes:
1> It turns out that to make this work I needed to incorporate SOLR-4013 and SOLR-3980.
I'd appreciate anyone looking at the synchronization I did around the member variable "loadingCores".
The intent here is to keep two threads from creating the same core at the same time.
1a> I'm assuming that there is exactly one CoreContainer per JVM. Otherwise I don't understand
how any of the synchronization works on the member vars....
1b> Running a simple test things went all to hell in JMX stuff without the synchronization,
apparently the multiple thread problem shows up early and often.
1c> synchronization is always "interesting", so the more eyes the better.
1d> In particular, any good suggestions about bailing out of the sleep loop? Since cores
can take quite a while to warm, I'm having a hard time thinking of a good default. I suppose
another attribute where the provider is mentioned. There's no reason a custom provider has
to be present, so requiring a timeout from the provider doesn't seem workable.

2> I've implemented a trivial CoreDescriptorProvider for a PoC, it's at the bottom. It
pre-supposes you have 4 collections, the accompanying Solr.xml is also below.

3> I'm going to put this away for a couple of hours and come back to it with fresh eyes,
this copy is purely for the curious/critical...

*****sample custom descriptor provider
public class TestCoreContainerProvider implements CoreDescriptorProvider
{
  @Override
  public CoreDescriptor getDescriptor(CoreContainer container, String name) {
    if (! "collection2".equals(name) && ! "collection3".equals(name) && !
"collection4".equals(name)) return null;

    CoreDescriptor cd = new CoreDescriptor(container, name, name); // True hack because I
know the dirs are the same as the collection.
    return cd;
  }

  @Override
  public boolean isPersist(String s) {
    return false;
  }

  @Override
  public Collection<String> getCoreNames() {
    return new ArrayList<String>(Arrays.asList("collection2", "collection3", "collection4"));
  }
}

**** solr.xml. Note no ZK stuff.
<solr persistent="false" sharedLib="../../../../../blahblah/out/artifacts/provider_jar">

  <cores adminPath="/admin/cores" defaultCoreName="collection1" host="${host:}" hostPort="${jetty.port:}"
hostContext="${hostContext:}" zkClientTimeout="${zkClientTimeout:15000}" coreDescriptorProviderClass="blah.TestCoreContainerProvider">
    <core name="collection1" instanceDir="collection1" />
  </cores>
</solr>
                
> Support pluggable persistence/loading of solr.xml details
> ---------------------------------------------------------
>
>                 Key: SOLR-1306
>                 URL: https://issues.apache.org/jira/browse/SOLR-1306
>             Project: Solr
>          Issue Type: New Feature
>          Components: multicore
>            Reporter: Noble Paul
>            Assignee: Erick Erickson
>             Fix For: 4.1
>
>         Attachments: SOLR-1306.patch
>
>
> Persisting and loading details from one xml is fine if the no:of cores are small and
the no:of cores are few/fixed . If there are 10's of thousands of cores in a single box adding
a new core (with persistent=true) becomes very expensive because every core creation has to
write this huge xml. 
> Moreover , there is a good chance that the file gets corrupted and all the cores become
unusable . In that case I would prefer it to be stored in a centralized DB which is backed
up/replicated and all the information is available in a centralized location. 
> We may need to refactor CoreContainer to have a pluggable implementation which can load/persist
the details . The default implementation should write/read from/to solr.xml . And the class
should be pluggable as follows in solr.xml
> {code:xml}
> <solr>
>   <dataProvider class="com.foo.FooDataProvider" attr1="val1" attr2="val2"/>
> </solr>
> {code}
> There will be a new interface (or abstract class ) called SolrDataProvider which this
class must implement

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message