From user-return-26109-archive-asf-public=cust-asf.ponee.io@flink.apache.org Fri Feb 22 15:49:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 072D6180648 for ; Fri, 22 Feb 2019 16:49:03 +0100 (CET) Received: (qmail 73512 invoked by uid 500); 22 Feb 2019 15:49:02 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 73502 invoked by uid 99); 22 Feb 2019 15:49:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Feb 2019 15:49:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 57122C2824 for ; Fri, 22 Feb 2019 15:49:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.797 X-Spam-Level: * X-Spam-Status: No, score=1.797 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id d6EHgS6xG2O3 for ; Fri, 22 Feb 2019 15:49:00 +0000 (UTC) Received: from mail-ua1-f67.google.com (mail-ua1-f67.google.com [209.85.222.67]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 9CBD35F2C3 for ; Fri, 22 Feb 2019 15:49:00 +0000 (UTC) Received: by mail-ua1-f67.google.com with SMTP id q17so2383981uam.0 for ; Fri, 22 Feb 2019 07:49:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=m8mW1UZZMwmoJ7sPVigPxJ2YEpvYrGHyerIRBs/y4DM=; b=OJXHHdB4tEyYIGJpNSq+BId1b8Wj3YDh3vdGLkxCVvSJPck3mksxeWAlCeGqEIIKMS OVZSH4N/xLiuQPYrw51L8uAyzoi1yAV9KQmOpf3hQul4TGMW9NoPxqDh3lWe+oduAUKs M9/8TQAPYHO41eB7dLBQEbvQvWesXcWXh+UYfWaHrgjBJ4eSzU8cq3hUJT+id0iMBfTc i+5tkdPwIN+qUORGE4vZZE8LBJohETj4R9zGBNriMr3fEXa4+p1vfzK+xfyzPaeCUHNi h4PppXvxHpAM7RdRDwKkmgNUL8fxMz1ndmwwJUv7v2e2cehGNnI+tL/YguWsYjL7iOuZ rXiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=m8mW1UZZMwmoJ7sPVigPxJ2YEpvYrGHyerIRBs/y4DM=; b=cRYgP7Y9gs2YZl4znl207a4innctTKfm/dBLsWMaMHMWRN3TV7P+U2QctEehfnTTgv VnweDr8Zef2SX33FOpQnj3jsVzBAQ8gOeP2qiCdmF6Y/TJ1tQo2LE7W9NMQEwkWZRzOE Ua+PkR+3FGKMTsOcD+HkcBkFP9f/sGoz9U4BKG+QGoCK0sn05WSfjD8E3WrDf49orPMS mfo8XT/UBMbEo8fzkK/BfSQM8alHDUTJD1CtJpqFfr7XHwv83BBSDVOYUcxnMk8H4VL8 uEtXyIqH4M9eA8w7X/i50qBtcjFDbkwglVrDgPTuoe5wfDBx0bQKlAAoqmM4NWtXkrm3 mhyA== X-Gm-Message-State: AHQUAubxElyiH8gEUprXHBzFDQwjc4huC5nNibNGhhn8Ct/8r3dAyKQc X+CEIfUk0Rvg4HoHWsIiTN7ZkaKvIp/0BymxzWs= X-Google-Smtp-Source: AHgI3IbqigJLW9tOgj7WeefDqr/or3solt6tVkXmyIdB2YPKnREbSi2/zP/Pl2BGxiLHNLSPfJ9iGp+3f7LVapIhr5E= X-Received: by 2002:ab0:6299:: with SMTP id z25mr2562925uao.69.1550850540037; Fri, 22 Feb 2019 07:49:00 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Durga Durga Date: Fri, 22 Feb 2019 07:48:49 -0800 Message-ID: Subject: Re: SinkFunction.Context To: Rong Rong Cc: user Content-Type: multipart/alternative; boundary="00000000000089ecac05827d87fe" --00000000000089ecac05827d87fe Content-Type: text/plain; charset="UTF-8" Excellent ! Thanks Rong.. On Thu, Feb 21, 2019 at 7:26 PM Rong Rong wrote: > Hi Durga, > > 1. currentProcessingTime: refers to this operator(SinkFunction)'s system > time at the moment of invoke > 1a. the time you are referring to as "flink window got the message" is the > currentProcessingTime() invoked at the window operator (which provided by > the WindowContext similar to this one [1]) > 2 currentWatermark: refers to the current watermark [2] received by this > operator(SinkFunction) > 3. timestamp: is actually the input record's event-time (this "input" is > referring to the input to the SinkFunction, not to the entire Flink > topology) > > Hope these help. > > -- > Rong > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#processwindowfunction > [2] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#event-time-and-watermarks > > On Thu, Feb 21, 2019 at 4:59 PM Durga Durga wrote: > >> >> HI Folks, >> >> Was following the documentation for >> >> >> https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.Context.html >> >> >> >> long currentProcessingTime >> >> () >> Returns the current processing time. >> long currentWatermark >> >> () >> Returns the current event-time watermark. >> Long >> >> timestamp >> >> () >> Returns the timestamp of the current input record or null if the element >> does not have an assigned timestamp. >> >> - CurrentProcessing time - is this is the Event Time ? i.e the time when >> the Event Occured ? (or) when the flink window got the message ?. >> >> - timeStamp - is this the time the record is persisted in to the sync ?. >> (or) the aggregated data timestamp ?. Say if I have 100 records in my time >> window - which time comes in to affect. >> >> - CurrentWaterMark - which time is this - the time the even occured - >> what will this value be - when there are 1000 records in my time window >> >> PS - We want to use some ID ( timestamp ) and associate with all the >> records that are persisted (aggregated) in a given time window - i.e if >> there are 1000 records aggregated - and they resulted in 10 aggregated >> records - we want to give these 10 aggregated records the same ID and we >> want to use one of the above time stamp >> >> Thanks much. >> > -- _Durga Deep --00000000000089ecac05827d87fe Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Excellent ! Thanks Rong..

<= br>
On Thu,= Feb 21, 2019 at 7:26 PM Rong Rong <walterddr@gmail.com> wrote:
Hi= Durga,

1. currentProcessingTime: refers to this operato= r(SinkFunction)'s system time at the moment of invoke
1a. the= time you are referring to as "flink window got the message" is t= he currentProcessingTime() invoked at the window operator (which provided b= y the WindowContext similar to this one [1])
2 currentWatermark: = refers to the current watermark [2] received by this operator(SinkFunction)=
3. timestamp: is actually the input record's event-time (thi= s "input" is referring to the input to the SinkFunction, not to t= he entire Flink topology)

Hope these help.

--
Rong

[2] https://ci.apache.org= /projects/flink/flink-docs-stable/dev/event_time.html#event-time-and-waterm= arks

On Thu, Feb 21, 2019 at 4:59 PM Durga Durga &= lt;durgadeep@gmail= .com> wrote:

HI Folks,
Was following the documentation for=C2=A0


=C2=A0 = =C2=A0 =C2=A0 =C2=A0=C2=A0
longcurrentProcessingTime()
Returns the current processing time.
<= code style=3D"font-family:"DejaVu Sans Mono",monospace;font-size:= 14px;padding-top:4px;margin-top:8px;line-height:1.4em">longcurrentWatermar= k()
Returns the current event-t= ime watermark.
Longtimestamp()
Returns the timestamp of the current= input record or=C2=A0null= =C2=A0if the element does not have an assigned timestamp.

- CurrentProcessing time - is= this is the Event Time ? i.e the time when the Event Occured ? (or) when t= he flink window got the message ?.

- timeStamp - i= s this the time the record is persisted in to the sync ?. (or) the aggregat= ed data timestamp ?. Say if I have 100 records in my=C2=A0 time window - wh= ich time comes in to affect.

- CurrentWaterMark - = which time is this - the time the even occured - what will this value be - = when there are 1000 records in my time window

PS -= We want to use some ID ( timestamp ) and associate with=C2=A0 all the reco= rds that are persisted (aggregated) in a given time window - i.e if there a= re 1000 records aggregated - and they resulted in 10 aggregated records - w= e want to give these 10 aggregated records the same ID and we want to use o= ne of the above time stamp

Thanks much.


--
_Durga Deep
--00000000000089ecac05827d87fe--