lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Courtade <courtadej...@gmail.com>
Subject Re: splitting shards on 4.7.2 with custom plugins
Date Wed, 26 Aug 2015 13:09:53 GMT
im looking at the clusterstate.json t see why it is doing this I really
dont understand it though...

{"collection1":{
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "state":"active",
            "base_url":"http://10.135.2.153:8981/solr",
            "core":"collection1",
            "node_name":"10.135.2.153:8981_solr",
            "leader":"true"},
          "core_node10":{
            "state":"active",
            "base_url":"http://10.135.2.153:8982/solr",
            "core":"collection1",
            "node_name":"10.135.2.153:8982_solr"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"inactive",
        "replicas":{
          "core_node9":{
            "state":"active",
            "base_url":"http://10.135.2.153:8984/solr",
            "core":"collection1",
            "node_name":"10.135.2.153:8984_solr",
            "leader":"true"},
          "core_node11":{
            "state":"active",
            "base_url":"http://10.135.2.153:8983/solr",
            "core":"collection1",
            "node_name":"10.135.2.153:8983_solr"}}},
      "shard1_1":{
        "range":null,
        "state":"active",
        "parent":null,
        "replicas":{
          "core_node6":{
            "state":"active",
            "base_url":"http://10.135.2.153:8981/solr",
            "core":"collection1_shard1_1_replica1",
            "node_name":"10.135.2.153:8981_solr",
            "leader":"true"},
          "core_node8":{
            "state":"active",
            "base_url":"http://10.135.2.153:8984/solr",
            "core":"collection1_shard1_1_replica2",
            "node_name":"10.135.2.153:8984_solr"}}},
      "shard1_0":{
        "range":null,
        "state":"active",
        "parent":null,
        "replicas":{
          "core_node5":{
            "state":"active",
            "base_url":"http://10.135.2.153:8981/solr",
            "core":"collection1_shard1_0_replica1",
            "node_name":"10.135.2.153:8981_solr",
            "leader":"true"},
          "core_node7":{
            "state":"active",
            "base_url":"http://10.135.2.153:8983/solr",
            "core":"collection1_shard1_0_replica2",
            "node_name":"10.135.2.153:8983_solr"}}},
      "shard2_0":{
        "range":"0-3fffffff",
        "state":"active",
        "replicas":{
          "core_node13":{
            "state":"active",
            "base_url":"http://10.135.2.153:8984/solr",
            "core":"collection1_shard2_0_replica1",
            "node_name":"10.135.2.153:8984_solr",
            "leader":"true"},
          "core_node14":{
            "state":"active",
            "base_url":"http://10.135.2.153:8982/solr",
            "core":"collection1_shard2_0_replica2",
            "node_name":"10.135.2.153:8982_solr"}}},
      "shard2_1":{
        "range":"40000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node12":{
            "state":"active",
            "base_url":"http://10.135.2.153:8984/solr",
            "core":"collection1_shard2_1_replica1",
            "node_name":"10.135.2.153:8984_solr",
            "leader":"true"},
          "core_node15":{
            "state":"active",
            "base_url":"http://10.135.2.153:8981/solr",
            "core":"collection1_shard2_1_replica2",
            "node_name":"10.135.2.153:8981_solr"}}}},
    "maxShardsPerNode":"1",
    "router":{"name":"compositeId"},
    "replicationFactor":"1",
    "autoCreated":"true"}}


--
Thanks,

Jeff Courtade
M: 240.507.6116

On Wed, Aug 26, 2015 at 8:44 AM, Jeff Courtade <courtadejeff@gmail.com>
wrote:

> Hi,
>
>
> So i got the shards too split. But they are very unbalanced.
>
>
>  7204922 total docs on the original collection
>
> shard1_0 numdocs 3661699
>
> shard1_1 numdocs 3543132
>
> shard2_0 numdocs 0
>
> shard2_1 numdcs 0
>
> Any ideas?
>
> This is what i had to do to get this to split with the custom libs
>
> I got shard1 to split successfully and it created replicas on the other
> servers in the cloud for the new shard/shards.
>
>
> This is the jist of it.
>
>
> When you split a shard solr creates a 2 new cores.
>
> When creating a core it uses the solr/solr.xml settings for classpath
> etc....
>
> This is why searches etc work fine and can find the opa plugins but when
> we called shardsplit it could not.
>
>
> I had to move the custom jars outside of the collection directory and add
> this to solr/solr.xml on the 4 nodes.
>
>
> info here .... https://wiki.apache.org/solr/Solr.xml%204.4%20and%20beyond
>
>
>
> <solr>
>
>
> <str name="sharedLib">${sharedLib:../lib}</str>
>
>
> when you restart you can see it in the log loading the jars form the new
> location.
>
>
>
> INFO  - 2015-08-25 23:40:52.297; org.apache.solr.core.CoreContainer;
> loading shared library: /opt/solr/solr-4.7.2/solr01/solr/../lib
>
> INFO  - 2015-08-25 23:40:52.298; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/commons-pool-1.6.jar' to
> classloader
>
> INFO  - 2015-08-25 23:40:52.298; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/query-processing-language-0.2-SNAPSHOT.jar'
> to classloader
>
> INFO  - 2015-08-25 23:40:52.299; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-continuation-8.1.10.v20130312.jar'
> to classloader
>
> INFO  - 2015-08-25 23:40:52.301; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/groovy-all-2.0.4.jar' to
> classloader
>
> INFO  - 2015-08-25 23:40:52.302; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/qpl-solr472-0.2-SNAPSHOT.jar'
> to classloader
>
> INFO  - 2015-08-25 23:40:52.302; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-jmx-8.1.10.v20130312.jar' to
> classloader
>
> INFO  - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-deploy-8.1.10.v20130312.jar' to
> classloader
>
> INFO  - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
> Adding 'file:/opt/solr/solr-4.7.2/solr01/lib/ext/' to classloader
>
> INFO  - 2015-08-25 23:40:52.303; org.apache.solr.core.SolrResourceLoader;
> Adding
> 'file:/opt/solr/solr-4.7.2/solr01/lib/jetty-xml-8.1.10.v20130312.jar' to
> classloader
>
> so I then ran the split and checked on it in the morning
>
>
> http://dj01.aws.narasearch.us:8981/solr/admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1
>
>
> it succeeded and created replicas.
>
> ls /opt/solr/solr-4.7.2/solr0*/solr/
>
> /opt/solr/solr-4.7.2/solr01/solr/:
> bin          collection1_shard1_0_replica1  README.txt  zoo.cfg
> collection1  collection1_shard1_1_replica1  solr.xml
>
> /opt/solr/solr-4.7.2/solr02/solr/:
> bin  collection1  README.txt  solr.xml  zoo.cfg
>
> /opt/solr/solr-4.7.2/solr03/solr/:
> bin  collection1  collection1_shard1_0_replica2  README.txt  solr.xml
>  zoo.cfg
>
> /opt/solr/solr-4.7.2/solr04/solr/:
> bin  collection1  collection1_shard1_1_replica2  README.txt  solr.xml
>  zoo.cfg
>
>
> and it actually distributed it
>
> [root@dj01 solr]# du -sh *
> 4.0K    bin
> 41G     collection1
> 18G     collection1_shard1_0_replica1
> 16G     collection1_shard1_1_replica1
> 4.0K    README.txt
> 4.0K    solr.xml
> 4.0K    zoo.cfg
> [root@dj01 solr]# du -sh
> /opt/solr/solr-4.7.2/solr04/solr/collection1_shard1_1_replica2
> 16G     /opt/solr/solr-4.7.2/solr04/solr/collection1_shard1_1_replica2
> [root@dj01 solr]# du -sh
> /opt/solr/solr-4.7.2/solr03/solr/collection1_shard1_0_replica2
> 18G     /opt/solr/solr-4.7.2/solr03/solr/collection1_shard1_0_replica2
>
>
> Jeff Courtade
> M: 240.507.6116
> On Aug 25, 2015 11:09 PM, "Anshum Gupta" <anshum@anshumgupta.net> wrote:
>
>> Can you elaborate a bit more on the setup, what do the custom plugins do,
>> what error do you get ? It seems like a classloader/classpath issue to me
>> which doesn't really relate to Shard splitting.
>>
>>
>> On Tue, Aug 25, 2015 at 7:59 PM, Jeff Courtade <courtadejeff@gmail.com>
>> wrote:
>>
>> > I am getting failures when trying too split shards on solr 4.2.7 with
>> > custom plugins.
>> >
>> > It fails regularily it cannot find the jar files for  plugins when
>> creating
>> > the new cores/shards.
>> >
>> > Ideas?
>> >
>> > --
>> > Thanks,
>> >
>> > Jeff Courtade
>> > M: 240.507.6116
>> >
>>
>>
>>
>> --
>> Anshum Gupta
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message