flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <leon_mcl...@tutanota.com>
Subject Elegantly sharing state in a streaming environment
Date Mon, 30 May 2016 11:41:30 GMT
Hello Flink team,

How can i partition and share static state among instances of a streaming 

I have a huge list of keys and values, which are used to filter tuples in a 
stream. The list does not change. Currently i am sharing the list with each 
operator instance via the constructor, although only a subset of the list is 
required per operator (the assignment of subset to operator instance is 
known). I cannot use DataSet based functions in a streaming execution 
environment to assign sub lists. I also cannot use DataStream based 
partitioning functions as the list is static, i.e. not a DataStream. The 
dilemma exists as i am mixing static (DataSet type) content with streaming 
content. Is there any other approach aside from using an additional tool 
(e.g. distributed cache)?

Thanks in advance.


View raw message