flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: BroadcastStream vs Broadcasted DataStream
Date Tue, 09 Oct 2018 13:55:15 GMT
Hi Pieter-Jan,

The second variant stores the elements of the broadcasted stream in operator (thus non-keyed)

On the differences: 

The Broadcast stream is not a keyed stream, so you are not in a keyed context, thus you have
no access to keyed state.
Given this, and assuming that you are implementing a CoProcessFunction, then your function
should also implement the 
ListCheckpointed interface and store the broadcasted data as ListState.

From a systems perspective, in case of rescaling, then your function should take care of removing
duplicated manually
as in the case of scaling down for example, the state of 2 nodes may end up on one. In addition,
given that elements are
re-distributed randomly, you may even end up with some elements missing one some nodes. This
would be more evident 
in case of scaling up. In this case, nodes will be missing data.

From an API perspective, assuming that the other stream is keyed and it has state, accessing
it in order to do some computation
would be impossible.


> On Oct 9, 2018, at 2:52 PM, Pieter-Jan Van Aeken <pieterjan.vanaeken@euranova.eu>
> Hi,
> I am not sure I fully understand the differences between doing something like
> dataStreamX.connect(dataStreamY.broadcast()).process(new CoProcessFunction{})
> and this
> dataStreamX.connect(dataStreamY.broadcast(descriptor).process(new BroadcastProcessFunction)
> Couldn't I manage the state exactly the same way using the first option? The only difference
I could find was in read/write permissions for different states. But other than that, the
difference in possibilities escapes me and I was hoping someone here could clarify.
> Kind regards,
> Pieter-Jan
> ___________________________
> Pieter-Jan Van Aeken
> Consultant - Data Engineer
> (M) +32 (0) 474 06 64 48
> Rue Emile Francqui, 4
> 1435 Mont-Saint-Guibert
> (T) +32 10 75 02 00
> euranova.eu <http://euranova.eu/>
> research.euranova.eu <http://research.euranova.eu/>
> ♻ Be green, keep it on the screen

View raw message