Mailing-List: contact dev-help@gearpump.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@gearpump.incubator.apache.org
Date: Fri, 15 Apr 2016 04:16:25 +0000 (UTC)
From: "Manu Zhang (JIRA)" <jira@apache.org>
To: dev@gearpump.incubator.apache.org
Message-ID: <JIRA.12959053.1460693680000.237378.1460693785430@Atlassian.JIRA>
In-Reply-To: <JIRA.12959053.1460693680000@Atlassian.JIRA>
References: <JIRA.12959053.1460693680000@Atlassian.JIRA>
 <JIRA.12959053.1460693680459@arcas>
Subject: [jira] [Updated] (GEARPUMP-32) Minimum clock of source Tasks maybe
 inaccurate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/GEARPUMP-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Manu Zhang updated GEARPUMP-32:
-------------------------------
    Description: 
Moved from [https://github.com/gearpump/gearpump/issues/1835] and reported by [Zhu Yueqian|https://github.com/yueqianzhu]

{quote}
Source tasks have not any upstreamClocks. So, startClock is the minimum of pending clocks when recover happen.

eg below:
source task1: timeStamp:15,not ACK, minClockValue maybe is 15(<= 15).
source task2: timeStamp:10,ACKed, minClockValue maybe is Long.MaxValue
when recover happen,startClock maybe is 15. where is the data between 10 to 15 at task2?
{quote}

More context on this issue:

In Gearpump, we maintain a global minimum clock tracked from a message's timestamp across all tasks. It means messages with timestamp before this clock have all been processed. An application will restart from this value on failure, and thus at-least-once message delivery could be guaranteed. 

The global minimum clock is the lower bound of all the Tasks' minimum clocks. 
For a task, the minimum clock is the lower of 

# upstream minimum clock
# a. the minimum timestamp of unacked messages
   b. Long.MaxValue if all messages have been acked.
 
Note that 2.b allows the global minimum clock to progress and it is almost safe since the clock is also bounded by the upstream minimum clock. I said "almost safe" because a source task has no upstream but we assume the upstream minimum clock is Long.MaxValue. Thus, the scenario described by Zhu Yueqian could happen and breaks at-least-once guarantee. 


  was:
Moved from [https://github.com/gearpump/gearpump/issues/1835] and reported by [Zhu Yueqian|https://github.com/yueqianzhu]

{quote}
Source tasks have not any upstreamClocks. So, startClock is the minimum of pending clocks when recover happen.

eg below:
source task1: timeStamp:15,not ACK, minClockValue maybe is 15(<= 15).
source task2: timeStamp:10,ACKed, minClockValue maybe is Long.MaxValue
when recover happen,startClock maybe is 15. where is the data between 10 to 15 at task2?
{quote}

More context on this issue:

In Gearpump, we maintain a global minimum clock tracked from a message's timestamp across all tasks. It means messages with timestamp before this clock have all been processed. An application will restart from this value on failure, and thus at-least-once message delivery could be guaranteed. 

The global minimum clock is the lower bound of all the Tasks' minimum clocks. 
For a task, the minimum clock is the lower of 

  1. upstream minimum clock
  2. a. the minimum timestamp of unacked messages
      b. Long.MaxValue if all messages have been acked.
 
Note that 2.b allows the global minimum clock to progress and it is almost safe since the clock is also bounded by the upstream minimum clock. I said "almost safe" because a source task has no upstream but we assume the upstream minimum clock is Long.MaxValue. Thus, the scenario described by Zhu Yueqian could happen and breaks at-least-once guarantee. 


> Minimum clock of source Tasks maybe inaccurate
> ----------------------------------------------
>
>                 Key: GEARPUMP-32
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-32
>             Project: Apache Gearpump
>          Issue Type: Bug
>          Components: streaming
>    Affects Versions: 0.8.0
>            Reporter: Manu Zhang
>            Assignee: Manu Zhang
>
> Moved from [https://github.com/gearpump/gearpump/issues/1835] and reported by [Zhu Yueqian|https://github.com/yueqianzhu]
> {quote}
> Source tasks have not any upstreamClocks. So, startClock is the minimum of pending clocks when recover happen.
> eg below:
> source task1: timeStamp:15,not ACK, minClockValue maybe is 15(<= 15).
> source task2: timeStamp:10,ACKed, minClockValue maybe is Long.MaxValue
> when recover happen,startClock maybe is 15. where is the data between 10 to 15 at task2?
> {quote}
> More context on this issue:
> In Gearpump, we maintain a global minimum clock tracked from a message's timestamp across all tasks. It means messages with timestamp before this clock have all been processed. An application will restart from this value on failure, and thus at-least-once message delivery could be guaranteed. 
> The global minimum clock is the lower bound of all the Tasks' minimum clocks. 
> For a task, the minimum clock is the lower of 
> # upstream minimum clock
> # a. the minimum timestamp of unacked messages
>    b. Long.MaxValue if all messages have been acked.
>  
> Note that 2.b allows the global minimum clock to progress and it is almost safe since the clock is also bounded by the upstream minimum clock. I said "almost safe" because a source task has no upstream but we assume the upstream minimum clock is Long.MaxValue. Thus, the scenario described by Zhu Yueqian could happen and breaks at-least-once guarantee. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)