flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Govindarajan Srinivasaraghavan <govindragh...@gmail.com>
Subject Dynamic Scaling
Date Fri, 23 Dec 2016 19:11:52 GMT

We have a computation heavy streaming flink job which will be processing
around 100 million message at peak time and around 1 million messages in
non peak time. We need the capability to dynamically scale so that the
computation operator can scale up and down during high or low work loads
respectively without restarting the job in order to lower the machine costs.

Is there an ETA on when the rescaling a single operator without restart
feature will be released?

Is it possible to auto scale one of the operators with docker swarm or
Amazon ECS auto scaling based on kafka consumer lag or cpu consumption? If
so can I get some documentation or steps to achieve this behaviour.

Also is there any document on what are the tasks of a job manager apart
from scheduling and reporting status?

Since there is just one job manager we just wanted to check if there would
be any potential scaling limitations as the processing capacity increases.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message