lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hendrik Haddorp <>
Subject Re: ClusterStateMutator
Date Thu, 05 Jan 2017 11:42:57 GMT
Right, I had to do that multiple times already when I restarted nodes 
during collection creation. In such cases I was left with data in the 
clusterstate.json, which at least on 6.2.1, blocked further collection 
creations. Once manually deleted or set to {} collection creation worked 

Setting legacyCloud=false looks good. I don't get anything in 
clusterstate.json anymore and no old collections show up after a node 
restarts. I could also confirm what Shalin said, that state format 2 is 
used by default. Only if I explicitly set state format to 1 I see data 
in clusterstate.json during the collection creation. Just the Solr admin 
UI is now showing "SolrCore Initialization Failures" pointing to none 
existing replicas. I assume that happens when Solr starts up and finds 
data for a core that does not exist in ZK anymore. How would one clean 
up this issue? Beside that the some replicas can still end up broken if 
the node restarts in the wrong time. I currently have one replica marked 
as down and one as gone. So far I was however always able to manually 
replace these replicas to resolve this state. So in general this looks 
quite good now. Guess I will still need to find a way to make sure that 
I don't restart a node during collection creation :-(

On 05.01.2017 02:33, Erick Erickson wrote:
> Let us know how it goes. You'll probably want to remove the _contents_
> of clusterstate.json and just leave it as a pair of brackets , i.e. {}
> if for no other reason than it's confusing.
> Times past the node needed to be there even if empty. Although I just
> tried removing it completely on 6x and I was able to start Solr, part
> of the startup process recreates it as an empty node, just a pair of
> braces.
> Best,
> Erick
> On Wed, Jan 4, 2017 at 1:22 PM, Hendrik Haddorp <> wrote:
>> Hi Erik,
>> I have actually also seen that behavior already. So will check what
>> happens when I set that property.
>> I still believe I'm getting the clusterstate.json set already before the
>> node comes up again. But I will try to verify that further tomorrow.
>> thanks,
>> Hendrik
>> On 04/01/17 22:10, Erick Erickson wrote:
>>> Hendrik:
>>> Historically in 4.x, there was code that would reconstruct the
>>> clusterstate.json code. So you would see "deleted" collections come
>>> back. One scenario was:
>>> - Have a Solr node offline that had a replica for a collection.
>>> - Delete that collection
>>> - Bring the node back
>>> - It would register itself in clusterstate.json.
>>> So my guess is that something like this is going on and you're getting
>>> a clusterstate.json that's reconstructed (and possibly not complete).
>>> You can avoid this by specifying legacyCloud=false clusterprop
>>> Kind of a shot in the dark...
>>> Erick
>>> On Wed, Jan 4, 2017 at 11:12 AM, Hendrik Haddorp
>>> <> wrote:
>>>> You are right, the code looks like it. But why did I then see collection
>>>> data in the clusterstate.json file? If version 1 is not used I would
>>>> assume that no data ends up in there. When explicitly setting the state
>>>> format 2 the system seemed to behave differently. And if the code always
>>>> uses version 2 shouldn't the default in that line be changed accordingly?
>>>> On 04/01/17 16:41, Shalin Shekhar Mangar wrote:
>>>>> Actually the state format defaults to 2 since many releases (all of
>>>>> 6.x at least). This default is enforced in CollectionsHandler much
>>>>> before the code in ClusterStateMutator is executed.
>>>>> On Wed, Jan 4, 2017 at 6:16 PM, Hendrik Haddorp <>
>>>>>> Hi,
>>>>>> in
>>>>>> solr-6.3.0/solr/core/src/java/org/apache/solr/cloud/overseer/
>>>>>> there is the following code starting line 107:
>>>>>> //TODO default to 2; but need to debug why BasicDistributedZk2Test
>>>>>> early on
>>>>>>      String znode = message.getInt(DocCollection.STATE_FORMAT, 1)
== 1 ? null
>>>>>>          : ZkStateReader.getCollectionPath(cName);
>>>>>> Any if that will be changed to default to version 2 anytime soon?
>>>>>> thanks,
>>>>>> Hendrik

View raw message