lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-5311) Avoid registering replicas which are removed
Date Tue, 31 Dec 2013 03:43:50 GMT

    [ https://issues.apache.org/jira/browse/SOLR-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859292#comment-13859292
] 

Mark Miller edited comment on SOLR-5311 at 12/31/13 3:42 AM:
-------------------------------------------------------------

This is part of a larger direction we have been working towards, which is essentially making
ZooKeeper the truth.

With the current SolrCloud, you cannot do this like you have. The Core Admin API, and pre
configured cores, and the collections API all need to work in concert. That makes this approach
a complete no go right now.

The path I have been working towards with the Collections API is a new mode where everything
is handled by the Collections API. In this case, it will not be valid for users to try and
mess with things at a per core level.

In this way, the cluster can truly match the truth in ZooKeeper because both the nodes and
the Overseer can work together to keep the truth enforced. This includes things like being
able to easily change the replication factor for a collection, or add a new host that automatically
gets used depending on your settings. You should not have to address individual nodes to manage
collections, nor should your replicationFactor stop mattering simply because you added another
core via core admin. To me, this is the ultimate cloud situation. The system needs full control.
We can add the ability to override or prefer certain things, but in general, we want to get
to the point of optionally have the cluster mostly managed for you given some simple directectives.
Of course, I think it should be implemented as a bunch of option features. You should also
be easy to really lock things done unless you manage things manually.

All of this requires we have a mode a user can decide to use (the collections api, perhaps
with an option for back compat) so that we are in control of everything. We know when a collection
is created and deleted - it won't be able to just pop back into existence. 

Until we have this special mode, the way that we had to build this, lots of historical reasons,
we currently have to support what we have with pre configured cores and core admin and the
collections api. This is silly form a user perspective though. It can all be done much nicer
with just a collections API that doesn't have to be directed to any single node. 

Doing what you want to do in a back compat way is not some simple fix. We have been working
towards this for a long time now - if you could just slap in a band aid and make it work like
this, I would have done it a long time go.


was (Author: markrmiller@gmail.com):
This is part of a larger direction we have been working towards, which is essentially making
ZooKeeper the truth.

With the current SolrCloud, you cannot do this like you have. The Core Admin API, and pre
configured cores, and the collections API all need to work in concert. That makes this approach
a complete no go right now.

The path I have been working towards with the Collections API is a new mode where everything
is handled by the Collections API. In this case, it will not be valid for users to try and
mess with things at a per core level.

In this way, the cluster can truly match the truth in ZooKeeper because both the nodes and
the Overseer can work together to keep the truth enforced. This includes things like being
able to easily change the replication factor for a collection, or a new host that automatically
gets used depending on your settings. You should not have to address individual nodes to manage
collections, nor should your replicationFactor stop mattering simply because you added another
core via core admin. To me, this is the ultimate cloud situation. The system needs full control.
We can add the ability to override or prefer certain things, but in general, we want to get
to the point of optionally have the cluster mostly managed for you given some simple directectives.
Of course, I think it should be implemented as a bunch of option features. You should also
be easy to really lock things done unless you manage things manually.

All of this requires we have a mode a user can decide to use (the collections api, perhaps
with an option for back compat) so that we are in control of everything. We know when a collection
is created and deleted - it won't be able to just pop back into existence. 

Until we have this special mode, the way that we had to build this, lots of historical reasons,
we currently have to support what we have with pre configured cores and core admin and the
collections api. This is silly form a user perspective though. It can all be done much nicer
with just a collections API that doesn't have to be directed to any single node. 

Doing what you want to do in a back compat way is not some simple fix. We have been working
towards this for a long time now - if you could just slap in a band aid and make it work like
this, I would have done it a long time go.

> Avoid registering replicas which are removed 
> ---------------------------------------------
>
>                 Key: SOLR-5311
>                 URL: https://issues.apache.org/jira/browse/SOLR-5311
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>            Reporter: Noble Paul
>            Assignee: Noble Paul
>             Fix For: 4.6, 5.0
>
>         Attachments: SOLR-5311.patch, SOLR-5311.patch, SOLR-5311.patch, SOLR-5311.patch,
SOLR-5311.patch, SOLR-5311.patch, SOLR-5311.patch
>
>
> If a replica is removed from the clusterstate and if it comes back up it should not be
allowed to register. 
> Each core ,when comes up, checks if it was already registered and if yes is it still
there. If not ,throw an error and do an unregister . If such a request come to overseer it
should ignore such a core



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message