apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tushar Gosavi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-532) New dynamically added operator does not start with correct windowId.
Date Thu, 22 Sep 2016 18:38:20 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514117#comment-15514117

Tushar Gosavi commented on APEXCORE-532:

If newly added operator is attached to some operator which was already exists in the DAG,
- It starts from activation checkpoint
- bufferserver subscriber use activation checkpoint id to set the initial baseSeconds (setup
of BufferServerSubscriber).
- when new data is received, we get partital windowId form tuple, or it with baseSeconds and
  construct full windowId.

baseSeconds is expected to set to correct value if operator starts from known checkpoint,
during initial dag deployment input operators sends resetWindow tuple, which sets baseSeconds
for all downstream operators. In this case operator was added in the end and was started
from activation checkpoint. but no resetWindow tuple was received to set its baseSeconds to
correct value.

In this case the starting checkpoint is different than windowId, may be we should use two
fields in
deploy info
- one for checkpoint from where to load the state
- starting baseSeconds which is taken from existing upstream to correctly set the windowIds
in absense of resetWindow tuple.

> New dynamically added operator does not start with correct windowId.
> --------------------------------------------------------------------
>                 Key: APEXCORE-532
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-532
>             Project: Apache Apex Core
>          Issue Type: Bug
>            Reporter: Tushar Gosavi
>            Priority: Critical
> During dynamic DAG change, If new operator is added and connected to existing operator,
it does not starts with correct windowId. The baseSeconds is set to 0 causing windowId management
problems at master effectively halting purge from buffer server.

This message was sent by Atlassian JIRA

View raw message