Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Govindarajan Srinivasaraghavan <govindraghvan@gmail.com>
Date: Fri, 23 Dec 2016 11:11:52 -0800
Message-ID: <CAP2Faz=vrOAzM9P9LHJ4WdetMTFjUecJQfkfM7j_-8xRB+qs_Q@mail.gmail.com>
Subject: Dynamic Scaling
To: dev@flink.apache.org, user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a113e6174a0057505445828e7
archived-at: Fri, 23 Dec 2016 19:12:04 -0000

--001a113e6174a0057505445828e7
Content-Type: text/plain; charset=UTF-8

Hi,

We have a computation heavy streaming flink job which will be processing
around 100 million message at peak time and around 1 million messages in
non peak time. We need the capability to dynamically scale so that the
computation operator can scale up and down during high or low work loads
respectively without restarting the job in order to lower the machine costs.

Is there an ETA on when the rescaling a single operator without restart
feature will be released?

Is it possible to auto scale one of the operators with docker swarm or
Amazon ECS auto scaling based on kafka consumer lag or cpu consumption? If
so can I get some documentation or steps to achieve this behaviour.

Also is there any document on what are the tasks of a job manager apart
from scheduling and reporting status?

Since there is just one job manager we just wanted to check if there would
be any potential scaling limitations as the processing capacity increases.

Thanks
Govind

--001a113e6174a0057505445828e7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>We have a computation heavy streami=
ng flink job which will be processing around 100 million message at peak ti=
me and around 1 million messages in non peak time. We need the capability t=
o dynamically scale so that the computation operator can scale up and down =
during high or low work loads respectively without restarting the job in or=
der to lower the machine costs.</div><div><br></div><div>Is there an ETA on=
 when the rescaling a single operator without restart feature will be relea=
sed?<br></div><div><br></div><div>Is it possible to auto scale one of the o=
perators with docker swarm or Amazon ECS auto scaling based on kafka consum=
er lag or cpu consumption? If so can I get some documentation or steps to a=
chieve this behaviour.</div><div><br></div><div>Also is there any document =
on what are the tasks of a job manager apart from scheduling and reporting =
status?=C2=A0</div><div><br></div><div>Since there is just one job manager =
we just wanted to check if there would be any potential scaling limitations=
 as the processing capacity increases.=C2=A0</div><div><br></div><div>Thank=
s</div><div>Govind</div><div><br></div></div>

--001a113e6174a0057505445828e7--