flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Govindarajan Srinivasaraghavan <govindragh...@gmail.com>
Subject Flink: How to handle external app configuration changes in flink
Date Mon, 26 Sep 2016 02:34:44 GMT
Hi,

My requirement is to stream millions of records in a day and it has huge
dependency on external configuration parameters. For example, a user can go
and change the required setting anytime in the web application and after
the change is made, the streaming has to happen with the new application
config parameters. These are app level configurations and we also have some
dynamic exclude parameters which each data has to be passed through and
filtered.

I see that flink doesn’t have global state which is shared across all task
managers and subtasks. Having a centralized cache is an option but for each
parameter I would have to read it from cache which will increase the
latency. Please advise on the better approach to handle these kind of
scenarios and how other applications are handling it. Thanks.

Mime
View raw message