Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1667A200BCF for ; Mon, 5 Dec 2016 13:56:33 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 14FD3160B17; Mon, 5 Dec 2016 12:56:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 11F85160AF9 for ; Mon, 5 Dec 2016 13:56:31 +0100 (CET) Received: (qmail 68273 invoked by uid 500); 5 Dec 2016 12:56:30 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 68264 invoked by uid 99); 5 Dec 2016 12:56:30 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Dec 2016 12:56:30 +0000 Received: from mail-io0-f171.google.com (mail-io0-f171.google.com [209.85.223.171]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 82CEB1A0044 for ; Mon, 5 Dec 2016 12:56:30 +0000 (UTC) Received: by mail-io0-f171.google.com with SMTP id a124so593875379ioe.2 for ; Mon, 05 Dec 2016 04:56:30 -0800 (PST) X-Gm-Message-State: AKaTC001TkBEqQLiPGLDyWrFdjKTXDTVnn1KPrtWisBt1dkgyMMDJlanZWXr8KOV1i0SJMgiC8bwwXTbXRmiZQ== X-Received: by 10.107.3.223 with SMTP id e92mr49835701ioi.113.1480942589860; Mon, 05 Dec 2016 04:56:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.48.18 with HTTP; Mon, 5 Dec 2016 04:56:29 -0800 (PST) In-Reply-To: References: <9942fad2-166b-15f5-9d50-252d5e201c5f@apache.org> <019D59A8-21CD-4E41-801F-1D50CC26413D@data-artisans.com> From: Stephan Ewen Date: Mon, 5 Dec 2016 13:56:29 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: microsecond resolution To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a113de9ca0063a30542e8d1c8 archived-at: Mon, 05 Dec 2016 12:56:33 -0000 --001a113de9ca0063a30542e8d1c8 Content-Type: text/plain; charset=UTF-8 @Jeff - good point about the docs. I think Kostas is right though - the event timestamps are up to the user's interpretation. The built-in window assigners interpret them as "Unix Epoch Millis", but you can define your own window assigners that interpret the timestamps differently. The system interprets them as also as Unix Epoch Millis when mixing event time and processing time (because processing time comes from System.currentTimeMillis()) So, you can "re-interpret" them by using custom window assigners and not using processing time. If you want to use a processing time like component, I'd suggest to incorporate that in your watermark generator. Stephan On Mon, Dec 5, 2016 at 1:05 PM, jeff jacobson < jeff.harold.jacobson@gmail.com> wrote: > Thanks Kostas. So if we're comfortable treating timestamps as longs (and > doing conversions to human readable time at our application level), we can > use CEP, ML lib etc. in addition to all basic Flink functions? That's great > news? > > To Matthias's point, *why then does the following not read "**Both > timestamps and watermarks are specified as longs"?* Before I go headlong > into developing on Flink, I just want to be sure I'm covered here. Again, > thanks. (The Youtube videos from FlinkForward are also great, btw. > Incredibly impressed with Data Artisans.) > > > https://ci.apache.org/projects/flink/flink-docs-release-1.1/ > apis/streaming/event_timestamps_watermarks.html#assigning-timestamps > > "Both timestamps and watermarks are specified as milliseconds since the Java > epoch of 1970-01-01T00:00:00Z." > > > > On Mon, Dec 5, 2016 at 4:57 AM, Kostas Kloudas < > k.kloudas@data-artisans.com> wrote: > >> Hi Jeff, >> >> Actually in Flink timestamps are simple longs. >> This means that you can assign anything you want as a timestamp, as long >> as it fits in a long. >> >> Hope this helps and if not, we can discuss to see if we can find a >> solution that >> fits your needs together. >> >> Cheers, >> Kostas >> >> On Dec 4, 2016, at 11:39 PM, jeff jacobson > m> wrote: >> >> Wow. Really? Is there a way to do micros? A hack? A Jira story? Most >> (all?) U.S. equity and European futures, options, and stock markets >> timestamp in microseconds. This makes Flink unusable for a massive industry >> vertical. To the extent lower-frequency time-series data is being used >> (e.g. end of data prices), stream processing is kind of overkill. Love >> everything I've read about Flink...there's got to be a way to make this >> work, no? >> >> On Sun, Dec 4, 2016 at 5:27 PM, Matthias J. Sax wrote: >> >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA512 >>> >>> Oh. My bad... Did not read your question carefully enough. >>> >>> Than the answer is no, it does not support microseconds (only >>> milliseconds). >>> >>> - -Matthias >>> >>> >>> On 12/4/16 2:22 PM, jeff jacobson wrote: >>> > Sorry if I'm missing something. That link mentions milliseconds, >>> > no? My question is whether or not I can specify microseconds where >>> > 1000microseconds = 1millisecond. Thanks! >>> > >>> > On Sun, Dec 4, 2016 at 5:05 PM, Matthias J. Sax >> > > wrote: >>> > >>> > Yes. It does. >>> > >>> > See: >>> > https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/strea >>> ming/event_timestamps_watermarks.html#assigning-timestamps >>> >>> > >>> > >>> >> ing/event_timestamps_watermarks.html#assigning-timestamps >>> >>> > >>> > >>> > "Both timestamps and watermarks are specified as millliseconds >>> > since the Java epoch of 1970-01-01T00:00:00Z." >>> > >>> > >>> > >>> > -Matthias >>> > >>> > >>> > On 12/04/2016 10:57 AM, jeff jacobson wrote: >>> >> I've sourced stackoverflow, the docs, and the web but I can't >>> >> figure out: does flink support microsecond timestamp resolution? >>> >> Thanks! >>> > >>> > >>> -----BEGIN PGP SIGNATURE----- >>> Comment: GPGTools - https://gpgtools.org >>> >>> iQIYBAEBCgAGBQJYRJhUAAoJELz8Z8hxAGOiNKoP32ChGeNd7N8Zco2q6lsu+Hxd >>> JZq62ey3wTrIUS+3oRlILwnu81cViQHtMMVBly3+YnqB85gNiaEUxEQTQCdKPl8G >>> AqxoFIkMcrKGzwGXigKnCAoVIiyuPeNuhY1d1yv4rWrkt7qb0lCC02Xoq1C0hoS6 >>> Stwk62GXmNRXPYpyjnSq/iAIMbjWaU+ZU0t4V3J8loroNuJ5QcUsJLfRXeo3/5ho >>> f42L+IANyB5K7vnTxNZYyf5ShNVbTY9/iFaviluxrCNztqGTo7CxMpcyWyMS3wcF >>> ycXcq/daB+guEJpW0sm4JtMPSsQ/kN99c/ig3t0HX1kDV7xrDDSF2qPvbYOWF38n >>> omTr7RY3YRFi5LOKvBGa96Aw5UYjMddjcqozWId6xgdXfvz6RUeJCWa9RW8I6ptg >>> 8TaJpM2WgDJMgMuzdl8dDv65l78DkLlNlNo53O66b/9Pt78P75KNjj8naD5kkj4C >>> i9amwnUNNEnZucA2/1vhzr6cVSzrzBLL7juVj0VmABZo4itUZjjR0UkN7MB+ioWU >>> trNhaXgE6EP/160n6D0/NUu02prm3jq8mK6gu9lZFWGbAeCUcch+CbvWSaiXAw3H >>> BOieCsgZD1wfXQJ3wEmnqj/YP94uDlx1IjynskDevjk6OIyIysbBSIqgsUK6fvQ8 >>> ztXO6ls7ARMOBmA= >>> =/O+Q >>> -----END PGP SIGNATURE----- >>> >> >> >> > --001a113de9ca0063a30542e8d1c8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
@Jeff - good point about the docs.

I th= ink Kostas is right though - the event timestamps are up to the user's = interpretation.

The built-in window assigners inte= rpret them as "Unix Epoch Millis", but you can define your own wi= ndow assigners that interpret the timestamps differently.
The= system interprets them as also as Unix Epoch Millis when mixing event time= and processing time (because processing time comes from System.currentTime= Millis())

So, you can "re-interpret"= them by using custom window assigners and not using processing time.
If you want to use a processing time like component, I'd sugge= st to incorporate that in your watermark generator.

Stephan


On Mon, Dec 5, 2016 at 1:05 PM, jeff jacobson <= span dir=3D"ltr"><jeff.harold.jacobson@gmail.com> wrote:
Thanks Kostas. So if we're c= omfortable treating timestamps as longs (and doing conversions to human rea= dable time at our application level), we can use CEP, ML lib etc. in additi= on to all basic Flink functions? That's great news?

To Matthias&= #39;s point, why then does the following not read "Both timestamps and watermarks are specified as lo= ngs"? Before I go headlong into developing on Flink, I just want t= o be sure I'm covered here. Again, thanks. (The Youtube videos from Fli= nkForward are also great, btw. Incredibly impressed with Data Artisans.)


https://ci.apache.org/projects/flink/flink-docs= -release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps

"Both timestamps and wate= rmarks are specified as milliseconds since the=C2=A0Java epoch of 1970-01-01T00:00:00Z."



On Mon, D= ec 5, 2016 at 4:57 AM, Kostas Kloudas <k.kloudas@data-artisans.c= om> wrote:
Hi Jeff,

Actually in Flink timestamp= s are simple longs.
This means that you can assign anything you w= ant as a timestamp, as long as it fits in a long.

= Hope this helps and if not, we can discuss to see if we can find a solution= that=C2=A0
fits your needs together.

Ch= eers,
Kostas

On Dec 4, 2016, at 11:= 39 PM, jeff jacobson <jeff.harold.jacobson@gmail.com> wrote:
<= br class=3D"m_7606682385119573266m_165017948118429687Apple-interchange-newl= ine">
Wow. Really? Is there a way to do micros? A hack= ? A Jira story? Most (all?) U.S. equity and European futures, options, and = stock markets timestamp in microseconds. This makes Flink unusable for a ma= ssive industry vertical. To the extent lower-frequency time-series data is = being used (e.g. end of data prices), stream processing is kind of overkill= . Love everything I've read about Flink...there's got to be a way t= o make this work, no?

On Sun, Dec 4, 2016 at 5:27 PM, Matthias J. Sax <mjsax@apache.org= > wrote:
-----BEGIN PGP SIG= NED MESSAGE-----
Hash: SHA512

Oh. My bad... Did not read your question carefully enough.

Than the answer is no, it does not support microseconds (only
milliseconds).

- -Matthias


On 12/4/16 2:22 PM, jeff jacobson wrote:
> Sorry if I'm missing something. That link mentions milliseconds, > no? My question is whether or not I can specify microseconds where
> 1000microseconds =3D 1millisecond. Thanks!
>
> On Sun, Dec 4, 2016 at 5:05 PM, Matthias J. Sax <mjsax@apache.org
> <mailto:mjsax@apache.org>> wrote:
>
> Yes. It does.
>
> See:
> https://ci.apache.org/projects/flink= /flink-docs-release-1.1/apis/strea
ming/event_timestamps_watermarks.html#assigning-timestamps
>
>
<https://ci.apache.org/projects/= flink/flink-docs-release-1.1/apis/stream
ing/event_timestamps_watermarks.html#assigning-timestamps>
>
> "Both timestamps and watermarks are specified as millliseconds > since the Java epoch of 1970-01-01T00:00:00Z."
>
>
>
> -Matthias
>
>
> On 12/04/2016 10:57 AM, jeff jacobson wrote:
>> I've sourced stackoverflow, the docs, and the web but I can= 9;t
>> figure out: does flink support microsecond timestamp resolution? >> Thanks!
>
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIYBAEBCgAGBQJYRJhUAAoJELz8Z8hxAGOiNKoP32ChGeNd7N8Zco2q6lsu+Hxd<= br> JZq62ey3wTrIUS+3oRlILwnu81cViQHtMMVBly3+YnqB85gNiaEUxEQTQCdKPl8G<= br> AqxoFIkMcrKGzwGXigKnCAoVIiyuPeNuhY1d1yv4rWrkt7qb0lCC02Xoq1C0hoS6<= br> Stwk62GXmNRXPYpyjnSq/iAIMbjWaU+ZU0t4V3J8loroNuJ5QcUsJLfRXeo3/5ho<= br> f42L+IANyB5K7vnTxNZYyf5ShNVbTY9/iFaviluxrCNztqGTo7CxMpcyWyMS3wcF<= br> ycXcq/daB+guEJpW0sm4JtMPSsQ/kN99c/ig3t0HX1kDV7xrDDSF2qPvbYOWF38n<= br> omTr7RY3YRFi5LOKvBGa96Aw5UYjMddjcqozWId6xgdXfvz6RUeJCWa9RW8I6ptg<= br> 8TaJpM2WgDJMgMuzdl8dDv65l78DkLlNlNo53O66b/9Pt78P75KNjj8naD5kkj4C<= br> i9amwnUNNEnZucA2/1vhzr6cVSzrzBLL7juVj0VmABZo4itUZjjR0UkN7MB+ioWU<= br> trNhaXgE6EP/160n6D0/NUu02prm3jq8mK6gu9lZFWGbAeCUcch+CbvWSaiXAw3H<= br> BOieCsgZD1wfXQJ3wEmnqj/YP94uDlx1IjynskDevjk6OIyIysbBSIqgsUK6fvQ8<= br> ztXO6ls7ARMOBmA=3D
=3D/O+Q
-----END PGP SIGNATURE-----



--001a113de9ca0063a30542e8d1c8--