flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: Lookup from state
Date Mon, 17 Sep 2018 14:29:23 GMT
Hi Taher,

To understand your use case, you have something like the following:

stream1.keyBy(…)
	.connect(stream2.keyBy(…))
	.window(…).apply(MyWindowFunction)

and you want from within the MyWindowFunction to access the state for a FIRED window when
a late element arrives for that key and window?

In this case, it depends what you want to do. 

First of all, with allowed lateness = 0, after the outofOrderness expires, then the state
is no longer there, so you cannot access it.

If you set the allowedLateness to > 0, then the state will be there until the allowed lateness
expires (see [1]), but in order to help with your use case,
it would help if you shared a bit more details about it. The reason is that form within the
MyWindowFunction, and depending on if 
you use apply() or reduce() or aggregate() or if you use an evictor, there are specific things
that you can do, and things that you cannot do.

Finally, for more control you may need to have a look at the CoProcessFunction in [2].

I hope this helps,
Kostas

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html>
[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html>

> On Sep 17, 2018, at 3:15 PM, Taher Koitawala <taher.koitawala@gslab.com> wrote:
> 
> Hi All,
>          As per my knowledge, all windowing operators in flink are stateful. So let's
say I have 2 streams, Stream1 and Stream2.
> 
> Stream1 and Stream2 are aggregated over some key and which is then windowed on EventTime.
So record X from Stream1 reaches flink on time, however, record X' from Stream2 reaches after
maxOutOfOrderness time has passed. In this scenario as per my knowledge. X will be maintained
in the flink state. However, when X' comes, how do I do a lookup for X from the flink state
and carry on the further aggregation or whatever I want to do.
> 
> 
> 
> Regards,
> Taher Koitawala
> GS Lab Pune
> +91 8407979163


Mime
View raw message