Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 18 Dec 2013 19:27:08 +0000 (UTC)
From: "Anshum Gupta (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.12680299.1384958593180.59484.1387394828017@arcas>
In-Reply-To: <JIRA.12680299.1384958593180@arcas>
References: <JIRA.12680299.1384958593180@arcas>
Subject: [jira] [Commented] (SOLR-5477) Async execution of
 OverseerCollectionProcessor tasks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852058#comment-13852058 ] 

Anshum Gupta commented on SOLR-5477:
------------------------------------

Having thought about it, here's another solution that also makes CoreAdmin calls async.

Adding a CoreAdmin API that also works on similar lines as that of Collection API for fetching back the request status. This will enable async Core Admin calls and avoid potential timeouts between the overseer and the cores.

Here's what all of the solution will look like:
* Async CollectionAPI calls for the end user which uses zk.
* Request status API for Collection level.
* CoreAdmin API to return request id as part of the response (if run in async mode?).
* CoreAdmin API to get the status of a submitted task (should solve the problem that you highlighted in particular).

I'm still debating between having even the CoreAdmin to use zk  (which means it'd only work in SolrCloud mode) or just have a local map of running taks. I'm anyways moving on with making the calls 'to' overseer async.

> Async execution of OverseerCollectionProcessor tasks
> ----------------------------------------------------
>
>                 Key: SOLR-5477
>                 URL: https://issues.apache.org/jira/browse/SOLR-5477
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Noble Paul
>            Assignee: Anshum Gupta
>
> Typical collection admin commands are long running and it is very common to have the requests get timed out.  It is more of a problem if the cluster is very large.Add an option to run these commands asynchronously
> add an extra param async=true for all collection commands
> the task is written to ZK and the caller is returned a task id. 
> as separate collection admin command will be added to poll the status of the task
> command=status&id=7657668909
> if id is not passed all running async tasks should be listed
> A separate queue is created to store in-process tasks . After the tasks are completed the queue entry is removed. OverSeerColectionProcessor will perform these tasks in multiple threads


--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org