apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tushar Gosavi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application
Date Mon, 01 Aug 2016 11:28:20 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401883#comment-15401883
] 

Tushar Gosavi commented on APEXCORE-408:
----------------------------------------

I have dome some initial prototype which allows stat listener to
specify dag changes, and the dag changes are applied asynchronously.

The changes involved are
- Add DagChangeSet object which is inherited from DAG, supporting
methods to remove operator and streams.

- The stat listener will return this object in Response, and platform
will apply changes specified in response to the DAG.

The Apex changes
https://github.com/apache/apex-core/compare/master...tushargosavi:scheduler?expand=1

The correspondign Demo application, which one operator monitors the
directory for files, and launch the wordcount DAG in
same application master when files are available.
https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/FileStatListenerSameDag.java

Let me know if this type of API is acceptable for modifying dag through java API.


> Ability to schedule Sub-DAG from running application
> ----------------------------------------------------
>
>                 Key: APEXCORE-408
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-408
>             Project: Apache Apex Core
>          Issue Type: Sub-task
>            Reporter: Thomas Weise
>            Assignee: Tushar Gosavi
>
> Today it is possible to add operators to a running application from the CLI/client. It
should be possible to do this from within the application also. This can be used to expand/remove
a Sub-DAG on demand, triggered by application specific logic. It will enable use cases such
as batch applications that perform multiple stages of processing where not all resources are
required at the same time or the logic of subsequent steps depends on the execution of previous
steps.    



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message