lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anshum Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-5477) Async execution of OverseerCollectionProcessor tasks
Date Wed, 18 Dec 2013 19:27:08 GMT

    [ https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852058#comment-13852058
] 

Anshum Gupta commented on SOLR-5477:
------------------------------------

Having thought about it, here's another solution that also makes CoreAdmin calls async.

Adding a CoreAdmin API that also works on similar lines as that of Collection API for fetching
back the request status. This will enable async Core Admin calls and avoid potential timeouts
between the overseer and the cores.

Here's what all of the solution will look like:
* Async CollectionAPI calls for the end user which uses zk.
* Request status API for Collection level.
* CoreAdmin API to return request id as part of the response (if run in async mode?).
* CoreAdmin API to get the status of a submitted task (should solve the problem that you highlighted
in particular).

I'm still debating between having even the CoreAdmin to use zk  (which means it'd only work
in SolrCloud mode) or just have a local map of running taks. I'm anyways moving on with making
the calls 'to' overseer async.

> Async execution of OverseerCollectionProcessor tasks
> ----------------------------------------------------
>
>                 Key: SOLR-5477
>                 URL: https://issues.apache.org/jira/browse/SOLR-5477
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Noble Paul
>            Assignee: Anshum Gupta
>
> Typical collection admin commands are long running and it is very common to have the
requests get timed out.  It is more of a problem if the cluster is very large.Add an option
to run these commands asynchronously
> add an extra param async=true for all collection commands
> the task is written to ZK and the caller is returned a task id. 
> as separate collection admin command will be added to poll the status of the task
> command=status&id=7657668909
> if id is not passed all running async tasks should be listed
> A separate queue is created to store in-process tasks . After the tasks are completed
the queue entry is removed. OverSeerColectionProcessor will perform these tasks in multiple
threads



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message