flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <leon_mcl...@tutanota.com>
Subject Elegantly sharing state in a streaming environment
Date Mon, 30 May 2016 11:41:30 GMT
Hello Flink team,

How can i partition and share static state among instances of a streaming 
operator?

I have a huge list of keys and values, which are used to filter tuples in a 
stream. The list does not change. Currently i am sharing the list with each 
operator instance via the constructor, although only a subset of the list is 
required per operator (the assignment of subset to operator instance is 
known). I cannot use DataSet based functions in a streaming execution 
environment to assign sub lists. I also cannot use DataStream based 
partitioning functions as the list is static, i.e. not a DataStream. The 
dilemma exists as i am mixing static (DataSet type) content with streaming 
content. Is there any other approach aside from using an additional tool 
(e.g. distributed cache)?

Thanks in advance.

Regards
Leon



Mime
View raw message