lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joel Bernstein (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-9417) Allow daemons to terminate when they finish iterating a topic
Date Wed, 19 Oct 2016 16:21:59 GMT

    [ https://issues.apache.org/jira/browse/SOLR-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589143#comment-15589143
] 

Joel Bernstein edited comment on SOLR-9417 at 10/19/16 4:21 PM:
----------------------------------------------------------------

Manual testing is looking good. The following expression worked well and the daemons terminated
when they finished processing the topic. So this the basic template of a batch job, that kicks
off then terminates when it's finished:

{code}
parallel(emails1, 
         workers=2, 
         sort="daemonOp asc",
         daemon(id="daemon1",
                terminate="true",
                update(emails, 
                          batchSize=250,
                          topic(checkpoints,
                             emails1,
                             q="*:*",
                             id="topic2",
                             initialCheckpoint=0,
                             partitionKeys="id",
                             fl="id, from, to, body"))))
{code}

The code above uses parallel workers to iterate over a topic and index the content to another
SolrCloud collection.


was (Author: joel.bernstein):
Manual testing is looking good. The following expression worked well and the daemons terminated
when they finished processing the topic. So this the basic template of a batch job, that kicks
off then terminates when it's finished:

{code}
parallel(emails1, 
         workers=2, 
         sort="daemonOp asc",
         daemon(id="daemon1",
                   terminate="true",
                   update(emails, 
                          batchSize=250,
                          topic(checkpoints,
                             emails1,
                             q="*:*",
                             id="topic2",
                             initialCheckpoint=0,
                             partitionKeys="id",
                             fl="id, from, to, body"))))
{code}

The code above uses parallel workers to iterate over a topic and index the content to another
SolrCloud collection.

> Allow daemons to terminate when they finish iterating a topic
> -------------------------------------------------------------
>
>                 Key: SOLR-9417
>                 URL: https://issues.apache.org/jira/browse/SOLR-9417
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Joel Bernstein
>            Assignee: Joel Bernstein
>             Fix For: 6.3
>
>         Attachments: SOLR-9417.patch, SOLR-9417.patch, SOLR-9417.patch
>
>
> The daemon expression currently runs until it's killed. This ticket will add a new *terminate*
parameter to the daemon expression that will allow the daemon to shut itself down when it's
finished processing a topic.
> There are a couple of small changes that need to be made to allow the daemon to terminate
on it's own:
> 1) The daemon will need to be passed the Map of all daemons in the /stream handler. This
will allow the DaemonStream to remove itself from the Map when it terminates.
> 2) Logic needs to be added for the daemon to exit it's run loop if the topic signals
it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can be used for this purpose.
If sleepMillis is greater then 0 then this signals a zero Tuple run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message