flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Govindarajan Srinivasaraghavan <govindragh...@gmail.com>
Subject Re: Dynamic Scaling
Date Tue, 27 Dec 2016 03:14:49 GMT
Hi All,

It would great if someone can help me with my questions. Appreciate all the help.

Thanks.

> On Dec 23, 2016, at 12:11 PM, Govindarajan Srinivasaraghavan <govindraghvan@gmail.com>
wrote:
> 
> Hi,
> 
> We have a computation heavy streaming flink job which will be processing around 100 million
message at peak time and around 1 million messages in non peak time. We need the capability
to dynamically scale so that the computation operator can scale up and down during high or
low work loads respectively without restarting the job in order to lower the machine costs.
> 
> Is there an ETA on when the rescaling a single operator without restart feature will
be released?
> 
> Is it possible to auto scale one of the operators with docker swarm or Amazon ECS auto
scaling based on kafka consumer lag or cpu consumption? If so can I get some documentation
or steps to achieve this behaviour.
> 
> Also is there any document on what are the tasks of a job manager apart from scheduling
and reporting status? 
> 
> Since there is just one job manager we just wanted to check if there would be any potential
scaling limitations as the processing capacity increases. 
> 
> Thanks
> Govind
> 

Mime
View raw message