beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kenneth Knowles <...@google.com>
Subject Re: Windowing a batch (python SDK 2.0.0)
Date Fri, 23 Jun 2017 14:02:24 GMT
Hello!

The behavior you are seeing is what makes something batch mode processing.
The essential definition of streaming mode processing is that you get
output before you have processed all the data.

Event time windowing does not control when computations occur - when you
window into FixedWindows of five seconds, this means your data will be
grouped according to the window that contains the timestamp on the event.
In streaming mode, this grouping will generally be output soon after the
watermark exceeds the end of the window, but this can be customized using
triggers.

Kenn

On Thu, Jun 22, 2017 at 10:13 AM, Morand, Sebastien <
sebastien.morand@veolia.com> wrote:

> Hi,
>
> I'm trying to window a batch, but whatever I try, the timestamp is not
> working, it's blocked by the group by :
> [image: Inline images 1]
>
> I want the transform-comine (the last on my screenshot) starts before the
> group_source has read all the files in input, and it's never happening.
> First it's reading all my files in the group by, and when it's over, it
> starts the transform-combine  ...
>
> I put a FixedWindow with 5 seconds, doesn't change anything, any way to do
> so?
>
> NB, JOB ID : 2017-06-22_10_03_47-4479358064592021427
>
> thanks by advance
>
> *S├ębastien MORAND*
> Team Lead Solution Architect
> Technology & Operations / Digital Factory
> Veolia - Group Information Systems & Technology (IS&T)
> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
> <+33%201%2085%2057%2071%2008>
> Bureau 0144C (Ouest)
> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
> *www.veolia.com <http://www.veolia.com>*
> <http://www.veolia.com>
> <https://www.facebook.com/veoliaenvironment/>
> <https://www.youtube.com/user/veoliaenvironnement>
> <https://www.linkedin.com/company/veolia-environnement>
> <https://twitter.com/veolia>
>
>
> ------------------------------------------------------------
> --------------------------------
> This e-mail transmission (message and any attached files) may contain
> information that is proprietary, privileged and/or confidential to Veolia
> Environnement and/or its affiliates and is intended exclusively for the
> person(s) to whom it is addressed. If you are not the intended recipient,
> please notify the sender by return e-mail and delete all copies of this
> e-mail, including all attachments. Unless expressly authorized, any use,
> disclosure, publication, retransmission or dissemination of this e-mail
> and/or of its attachments is strictly prohibited.
>
> Ce message electronique et ses fichiers attaches sont strictement
> confidentiels et peuvent contenir des elements dont Veolia Environnement
> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> message par erreur, merci de le retourner a son emetteur et de le detruire
> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
> publication, la distribution, ou la reproduction non expressement
> autorisees de ce message et de ses pieces attachees sont interdites.
> ------------------------------------------------------------
> --------------------------------
>

Mime
View raw message