flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Why would indefinitely growing state an issue for Flink while doing stream to stream joins?
Date Thu, 16 Jan 2020 15:06:28 GMT
Hi All,

The doc
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/joins.html#regular-joins
says
the following.

"However, this operation has an important implication: it requires to keep
both sides of the join input in Flinkā€™s state forever. Thus, the resource
usage will grow indefinitely as well, if one or both input tables are
continuously growing"

I wonder why this would be an issue especially when the state is stored in
RocksDB which in turn is backed by disk?

I have a use case where I might need to do stream-stream join or some
emulation of that across say 6 or more tables and I don't know for sure how
long I need to keep the state because a row today can join with a row a
year or two years from now. will that be an issue? do I need to think about
designing a solution in another way without using stream-stream join?

Thanks!

Mime
View raw message