apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-532) New dynamically added operator does not start with correct windowId.
Date Fri, 16 Sep 2016 13:06:20 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496291#comment-15496291
] 

ASF GitHub Bot commented on APEXCORE-532:
-----------------------------------------

GitHub user tushargosavi opened a pull request:

    https://github.com/apache/apex-core/pull/393

    **Review Only** Apexcore 408

    Pull request for dynamic dag modification through stats listener.  It provides following
    functionality
    
    - StatsListener can access the opearator name for easily detecting which opearator stats
are being processed.
    - StatsListener can create a instance of object through which it can submit dag modifications
to the engine.
    - StatsListener can return dag changes as a response to engine.
    - PlanModifier is modified to take a DAG and apply it on the existing running DAG and
deploy the changes.
    
    The following functionality is not working yet.
    
    - The new opearator does not start from the correct windowId (https://issues.apache.org/jira/browse/APEXCORE-532)
    - Relanched application failed to start when it was killed after dynamic dag modification.
    - There is no support for resuming operator from previous state when they were removed.
This could be achived through
      readig state through external storage on setup.
    - persist operator support is not present for newly added streams.
    - Not all parts are covered through tests.
    
    The demo application using the feature is available at
    https://github.com/tushargosavi/apex-dynamic-scheduling
    
    There are two variations of WordCount application. The first variation detects the presence
of
    new files start a disconnected DAG to process the data.
    (https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/WordCountApp.java)
    
    The second application (https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/ExtendApp.java),
    initially only one reader operator is running in the DAG, and provides pendingFiles as
auto-metric to stat listener.
    On detecting pending files it attaches splitter counter and output operator to the read
operator. Once files are processed the splitter, counter and
    output operators are removed and added back again if new data files are added into the
directory.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tushargosavi/incubator-apex-core APEXCORE-408

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/apex-core/pull/393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #393
    
----
commit 05fcca2c919e2828f8348e5e4ed60d87e044b5ac
Author: Tushar R. Gosavi <tushar@apache.org>
Date:   2016-08-08T10:24:41Z

    APEXCORE-496: Provide way to access dag elements for statslisteners.

commit 8cf49f28a7732787c2517a5b345e4690be517280
Author: Tushar R. Gosavi <tushar@apache.org>
Date:   2016-07-05T08:40:26Z

    APEXCORE-408: Dynamic dag changes in Apex through StatsListeners

----


> New dynamically added operator does not start with correct windowId.
> --------------------------------------------------------------------
>
>                 Key: APEXCORE-532
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-532
>             Project: Apache Apex Core
>          Issue Type: Bug
>            Reporter: Tushar Gosavi
>            Priority: Critical
>
> During dynamic DAG change, If new operator is added and connected to existing operator,
it does not starts with correct windowId. The baseSeconds is set to 0 causing windowId management
problems at master effectively halting purge from buffer server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message