Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 65E47200BAA for ; Wed, 12 Oct 2016 16:32:20 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 644FA160AD4; Wed, 12 Oct 2016 14:32:20 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DEA12160AD3 for ; Wed, 12 Oct 2016 16:32:18 +0200 (CEST) Received: (qmail 38003 invoked by uid 500); 12 Oct 2016 14:32:12 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 37992 invoked by uid 99); 12 Oct 2016 14:32:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Oct 2016 14:32:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 591461A06B2 for ; Wed, 12 Oct 2016 14:32:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.299 X-Spam-Level: * X-Spam-Status: No, score=1.299 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=data-artisans-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id j0DhLFIopXAa for ; Wed, 12 Oct 2016 14:32:09 +0000 (UTC) Received: from mail-qk0-f175.google.com (mail-qk0-f175.google.com [209.85.220.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 1221C5F1EE for ; Wed, 12 Oct 2016 14:32:09 +0000 (UTC) Received: by mail-qk0-f175.google.com with SMTP id o68so80245846qkf.3 for ; Wed, 12 Oct 2016 07:32:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=data-artisans-com.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=AiN8jfXzG6HaoBmiVpOspBiJgMfWhSC1ciligiedl1g=; b=1Lve+WdAtvu1Usa1aQqe2v+ECn+FbhEXKftuBOi3sTtW5N/fQrF97kV0S9RmYqhfrp NySnTsOs9YpSN22BGoXmj0r6J/yk7V6nWhBMs2NJLWo9wRsaawCSLtp7m2pMhuN+1Hbu pEiMYm4hsiYiyxvtwlaOJ1h49cAAZ7jG0i6MsfYOfoEnQRmprl0ooQd0Vw0g8ggmsK3O lq2nRpx6b0HWBDasucCcZpEJNOFURA4nxyfqs/ryl88Kvt4Trd6h6ZGaS7xD2j+ynY/8 pNpfqGoOFXASCgwi+0jnQEcSotodMOwZd7vdb5LiTPTM6HPEDQnFeL9qf+wUNf4k7kB7 cY5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=AiN8jfXzG6HaoBmiVpOspBiJgMfWhSC1ciligiedl1g=; b=DwjefRpqPe6H2V2gNs6/ITIoidIRNjiMrgHVgcZYTv9AR3p2+w8OC4XGD5eK+rMvBU E2rx/Z1g0GDdC73pZeHTGYzZrMd9RObA7RIPe+pxUiY7j1jTUWAF/OcxAdbNF4oSRkGl oklbC2zwDiuDnYzJoP/GG0Ubop0QAG0kd4x8RYQvcAQ4AS+j6G3RjhmutfgOtexCYMyd rgZOIe/ZD2Hsci5g9h9aeg6mRlfz1oIpdIaBzwqOpklQJVtvaSylfMJoz+uTUa/yR/aC N/tRHvA1/d2z83bwyFGthYp/PFwz20ouE6HWd2QSzX+YCIqUB2TvQhqY8BbRf/ZYqHro 38ZA== X-Gm-Message-State: AA6/9Rk4l6QtNfvz4B9+S56v4K1Ce2hXFZFfPwGcIPzC8QHfKmI2S5nzxi7nsXQ1i8kFGLou X-Received: by 10.194.170.8 with SMTP id ai8mr2066489wjc.118.1476282726189; Wed, 12 Oct 2016 07:32:06 -0700 (PDT) Received: from macklou.fritz.box (ip5b4035dc.dynamic.kabel-deutschland.de. [91.64.53.220]) by smtp.gmail.com with ESMTPSA id g9sm13380754wjk.25.2016.10.12.07.32.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Oct 2016 07:32:05 -0700 (PDT) From: Kostas Kloudas Message-Id: <6F53BF70-0BA0-4609-831E-865FF3B6471A@data-artisans.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_57F60391-0F0A-4B04-A308-2CE95167D718" Mime-Version: 1.0 (Mac OS X Mail 10.0 \(3226\)) Subject: Re: About Sliding window Date: Wed, 12 Oct 2016 16:32:03 +0200 In-Reply-To: Cc: "user@flink.apache.org" , Aljoscha Krettek To: Zhangrucong References: X-Mailer: Apple Mail (2.3226) archived-at: Wed, 12 Oct 2016 14:32:20 -0000 --Apple-Mail=_57F60391-0F0A-4B04-A308-2CE95167D718 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello, So in your previous figure (yesterday) when e3 arrives, also e2 should = be included in the result, right? In this case, I think that what you need is a Session window with gap = equal to your event aging duration and an evictor that evicts the elements that lag behind more than the gap = duration. The latter, the evictor that I am describing, is not currently supported = in Flink but there is an ongoing=20 discussion in the dev mailing list about it. So it is worth having a = look there and participate in the discussion. I also loop in Aljoscha in the discussion, in case he has another = solution that you can deploy right-away. Thanks, Kostas > On Oct 12, 2016, at 3:36 PM, Zhangrucong = wrote: >=20 > Hi Kostas: > It doesn=E2=80=99t matter. Can you see the picture? My = user case is: > =20 > 1=E3=80=81The events are coming according to the following order > > At 9:01 e1 is coming > At 9:02 e2 is coming > At 9:06 e3 is coming > At 9:08 e4 is coming > =20 > The time is system time. > =20 > 2=E3=80=81And event aging time is 5 minutes. > =20 > 3=E3=80=81 > At 9:01 e1 is coming, aged nothing, store e1,we count e1 and = send the result. > At 9:02 e2 is coming, aged nothing, store e2, We count e1 and = e2. and send the result. > At 9:06 e3 is coming, aged e1, store e3, we count e2 and e3, = and send the result. > At 9:08 e4 is coming, aged e2, store e4, we count e3 and e4, = and send the result. > =20 > =20 > I think I need a certain duration window. > =20 > Thank you very much! > =E5=8F=91=E4=BB=B6=E4=BA=BA: Kostas Kloudas = [mailto:k.kloudas@data-artisans.com]=20 > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B410=E6=9C=8812=E6=97=A5= 21:11 > =E6=94=B6=E4=BB=B6=E4=BA=BA: Zhangrucong > =E6=8A=84=E9=80=81: user@flink.apache.org > =E4=B8=BB=E9=A2=98: Re: About Sliding window > =20 > Hello again, > =20 > Sorry for the delay but I cannot really understand your use case. > Could you explain a bit more what do you mean by =E2=80=9Cout-of-date=E2= =80=9D event and =E2=80=9Caging=E2=80=9D an event? > =20 > Also your windows are of a certain duration or global? > =20 > Thanks, > Kostas > =20 > On Oct 11, 2016, at 3:04 PM, Zhangrucong > wrote: > =20 > Hi Kostas: > Thank you for your rapid response! > =20 > My use-case is that : > For every incoming event, we want to age the out-of-date event , count = the event in window and send the result. > =20 > For example: > The events are coming as flowing: > > =20 > We want flowing result: > > =20 > =20 > By the way, In StreamSQL API, in FILP11, It will realize row window. = It seems that the function of Slide Event-time row-window suits my = use-case. Does data stream API support row window? > =20 > Thanks ! > =20 > =E5=8F=91=E4=BB=B6=E4=BA=BA: Kostas Kloudas = [mailto:k.kloudas@data-artisans.com = ]=20 > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B410=E6=9C=8811=E6=97=A5= 19:38 > =E6=94=B6=E4=BB=B6=E4=BA=BA: user@flink.apache.org = > =E4=B8=BB=E9=A2=98: Re: About Sliding window > =20 > Hi Zhangrucong, > =20 > Sliding windows only support time-based slide.=20 > So your use-case is not supported out-of-the-box. > =20 > But, if you describe a bit more what you want to do,=20 > we may be able to find a way together to do your job using=20 > the currently offered functionality. > =20 > Kostas > =20 > On Oct 11, 2016, at 1:20 PM, Zhangrucong > wrote: > =20 > Hello everyone: > Now, I am want to use DataStream sliding window API. I look at the = API and I have a question, dose the sliding time window support sliding = by every incoming event? > =20 > Thanks in advance! > =20 > > =20 > --Apple-Mail=_57F60391-0F0A-4B04-A308-2CE95167D718 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hello,

So = in your previous figure (yesterday) when e3 arrives, also e2 should be = included in the result, right?

In this case, I think that what you = need is a Session window with gap equal to your event aging duration = and
an evictor that evicts the elements that lag = behind more than the gap duration.

The latter, the evictor that I am = describing, is not currently supported in Flink but there is an = ongoing 
discussion in the dev mailing list = about it. So it is worth having a look there and participate in the = discussion.

I = also loop in Aljoscha in the discussion, in case he has another solution = that you can deploy right-away.

Thanks,
Kostas

On = Oct 12, 2016, at 3:36 PM, Zhangrucong <zhangrucong@huawei.com> wrote:

Hi Kostas:
          &nb= sp; It doesn=E2=80=99t matter. Can you see the picture? My user case = is:
 
1=E3=80=81The events are coming according to the following order
<image004.png>
At 9:01 e1 is = coming
At 9:02 e2 is = coming
At 9:06 =  e3 is coming
At = 9:08   e4 is coming
 
The time is = system time.
 
2=E3=80=81And  event aging time is 5 minutes.
 
3=E3=80=81
       At 9:01 e1 is coming, = aged nothing, store e1,we count e1 and send the result.
       At 9:02 e2 is coming, =  aged nothing, store e2,  We count e1 and e2. and send the = result.
      At 9:06  e3 is coming, =  aged e1,  store e3, we count e2 and e3, and send the = result.
     At 9:08   e4 is coming, =  aged e2,  store e4, we count e3 and e4, and send the = result.
 
 
I think I need = a certain duration window.
 
Thank you very = much!
=E5=8F=91=E4=BB=B6=E4=BA=BA: Kostas Kloudas [mailto:k.kloudas@data-artisans.com] 
=E5=8F=91=E9=80=81=E6= =97=B6=E9=97=B4: 2016=E5=B9=B410=E6=9C=8812=E6=97=A5 21:11
=E6=94=B6=E4=BB=B6=E4=BA=BA: Zhangrucong
=E6=8A=84=E9=80=81: user@flink.apache.org
=E4=B8=BB=E9=A2=98: Re: About Sliding = window
 
Hello again,
 
Sorry = for the delay but I cannot really understand your use case.
Could you explain a bit more = what do you mean by =E2=80=9Cout-of-date=E2=80=9D event and =E2=80=9Caging= =E2=80=9D an event?
 
Also your windows are of a certain duration or global?
 
Thanks,
Kostas
 
On Oct 11, 2016, at 3:04 PM, = Zhangrucong <zhangrucong@huawei.com> wrote:
 
Hi Kostas:
Thank you for = your rapid response!
 
My use-case is = that :
For every incoming event, we want to age = the out-of-date event , count the event in window and send the = result.
 
For = example:
The events are coming as = flowing:
<image002.png>
 
We want = flowing result:
<image004.png>
 
 
By the way, In = StreamSQL API, in FILP11, It will realize row window. It seems that the = function of Slide Event-time row-window suits my use-case. Does data = stream API  support row window?
 
Thanks = !
 
=E5=8F=91=E4=BB=B6=E4=BA=BA: Kostas Kloudas [mailto:k.kloudas@data-artisans.com] 
=E5=8F=91=E9=80=81=E6= =97=B6=E9=97=B4: 2016=E5=B9=B410=E6=9C=8811=E6=97=A5 19:38
=E6=94=B6=E4=BB=B6=E4=BA= =BA: user@flink.apache.org
=E4=B8=BB=E9=A2=98: Re: About Sliding window
 
Hi Zhangrucong,
 
Sliding = windows only support time-based slide. 
So your = use-case is not supported out-of-the-box.
 
But, if you = describe a bit more what you want to do, 
we may be = able to find a way together to do your job using 
the = currently offered functionality.
 
Kostas
 
On Oct 11, 2016, at 1:20 PM, Zhangrucong <zhangrucong@huawei.com> wrote:
 
Hello = everyone:
  Now, I am want to use DataStream sliding window API. I = look at the API and I have a question, dose the sliding time window = support sliding by every incoming event?
 
Thanks in advance!
 
<oledata.mso>
 
<oledata.mso>
=
= --Apple-Mail=_57F60391-0F0A-4B04-A308-2CE95167D718--