apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-494) Window id of downstream operator is not moving after dynamic partition of upstream operator.
Date Thu, 21 Jul 2016 22:48:20 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388560#comment-15388560
] 

ASF GitHub Bot commented on APEXCORE-494:
-----------------------------------------

Github user vrozov commented on a diff in the pull request:

    https://github.com/apache/apex-core/pull/359#discussion_r71801328
  
    --- Diff: engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java ---
    @@ -895,8 +895,10 @@ private void redoPartitions(PMapping currentMapping, String note)
             addedPartitions.add(newPartition);
           } else {
             // check whether mapping was changed
    +        int currentPartitionsSize = mainPC.currentPartitions.size();
             for (DefaultPartition<Operator> pi : mainPC.currentPartitions) {
    -          if (pi == newPartition && pi.isModified()) {
    +          if (pi == newPartition && (pi.isModified() ||
    --- End diff --
    
    @tweise Should not redeploy of downstream being done only when downstream pipeline has
a unifier and only unifier needs to be redeployed for the pipeline connected to the current
partition? For example, if logical dag is defined as:
    ```
    A->B-C->Aggregator
    ```
    and it is partitioned to 2 pipelines starting with A:
    ```
    A1->B1->C1->
                 U->Aggregator
    A2->B2->C2->
    ```
    and A is repartitioned to 3:
    ```
    A1->B1->C1->
    A2->B2->C2->U->Aggregator
    A3->B3->C3->
    ```
    A1->B1->C1 and A2->B2->C2 don't need to be redeploed? 


> Window id of downstream operator is not moving after dynamic partition of upstream operator.
> --------------------------------------------------------------------------------------------
>
>                 Key: APEXCORE-494
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-494
>             Project: Apache Apex Core
>          Issue Type: Bug
>            Reporter: Chaitanya
>            Assignee: Chaitanya
>         Attachments: PhysicalDAGAfterKafkaRepartition.png, PhysicalDAGBeforeKafkaRepartition.png,
PhysicalOperatorAfterKafkaRepartition.png
>
>
> Created the following application to reproduce this issue.
> Application has the following DAG:
>    KafkaSinglePortStringInputOperator(Input) -> ConsoleOutputOperator(Output)
>    
>    I launched the application with below configuration:
> Kafka topic created with single partition and replication factor as 1.
> Partition Strategy: ONE_TO_ONE
>    Launched the application successfully. After some time, I increased the topic partitions
to 2. After re-partition, the window of down stream operator is not moving. By looking into
the app Physical DAG, it looks like there is an issue in construction of Physical DAG after
re-partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message