lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-3488) Create a Collections API for SolrCloud
Date Fri, 15 Jun 2012 14:36:44 GMT

    [ https://issues.apache.org/jira/browse/SOLR-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295689#comment-13295689
] 

Mark Miller commented on SOLR-3488:
-----------------------------------

I've got my first patch ready - still some things to address, but it currently does queue
based collection creation.

One thing I recently realized when I put some last minute pieces together is that I cannot
share the same Overseer queue that already exists - it will cause a deadlock as we wait for
states to be registered. Processing the queue with multiple threads still seemed scary if
you were to create a lot of collections at once - so it seems just safer to use a different
queue.

I'm still somewhat unsure about handing failures though - for the moment I'm simply adding
the job back onto the queue - this gets complicated quickly though. Especially if you add
in delete collection and it can fail. If you start putting commands back on the queue you
could have weird create/delete command retry reordering?

I also have not switched to requiring or respecting a replication factor - I was thinking
perhaps specifying nothing or -1 would give you what you have now? An infinite rep factor?
And we would enforce a lower rep factor if requested? For now I still require that you pass
a collection template and new nodes are created on the nodes that host the collection template.

I'm not sure how replication factor would be enforced though? The Oveerseer just periodically
prunes and adds given what it sees and what the rep factor is? Is that how failures should
be handled? Don't readd to the queue, just let the periodic job attempt to fix things later?


What if someone starts a new node with a new replicas pre configured in solr.xml? The Overseer
periodic job would simply remove them shortly thereafter if the rep factor was not high enough?

One issue with pruning at the moment is that unloading a core will not remove it's data dir.
We probably want to fix that for collection removal.

If we go too far down this path, it seems rebalancing starts to become important as well.

I've got some other thoughts and ideas to get down, but that is a start so I can gather some
feedback.

I have not yet integrated Tomasso's work, but will if we don't end up changing things much
from now.
                
> Create a Collections API for SolrCloud
> --------------------------------------
>
>                 Key: SOLR-3488
>                 URL: https://issues.apache.org/jira/browse/SOLR-3488
>             Project: Solr
>          Issue Type: New Feature
>          Components: SolrCloud
>            Reporter: Mark Miller
>            Assignee: Mark Miller
>         Attachments: SOLR-3488.patch, SOLR-3488.patch, SOLR-3488_2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message