beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kenneth Knowles <...@google.com>
Subject Re: How does Beam set up the bundle size in streaming mode (like Pub/Sub)?
Date Wed, 18 Oct 2017 13:59:03 GMT
Bundles are decidedly not windows, so let's keep the two terms separate. It
sounds like you are asking about bundles.

The bundle size is a performance tuning parameter and is arbitrarily chosen
arbitrarily and dynamically chosen by a runner. The runner chooses based on
its best effort to amortize @StartBundle/@FinishBundle operations across
multiple @ProcessElement/@OnTimer calls. Your code must yield correct
results for for any bundling - you should be implementing per-element
logic, where @StartBundle/@FinishBundle are implementation details.

Kenn

On Tue, Oct 17, 2017 at 5:37 PM, Derek Hao Hu <phoenixinter@gmail.com>
wrote:

> Hi,
>
> Is there any more detailed explanation on how Beam chooses the window size
> (bundle size) in streaming mode? It seems there is no clear answer in the
> [Beam Programming Guide](https://beam.apache.
> org/documentation/programming-guide/) and I can't find how PubsubIO
> implements this windowing strategy as well. :(
>
> Could someone kindly provide some pointers? Thanks!
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>

Mime
View raw message