flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sri hari kali charan Tummala <kali.tumm...@gmail.com>
Subject Re: Running flink on AWS ECS
Date Sat, 28 Sep 2019 13:04:35 GMT
Aws already has auto scale flink cluster it’s called Kinesis Data Analytics
just add your flink Jar to Kinesis Sql analytics that’s all , aws will auto
provision a flink cluster and do the admin part for you.

On Saturday, September 28, 2019, David Anderson <david@ververica.com> wrote:

> I believe there can be advantages and disadvantages in both
> directions. For example, fewer containers with multiple slots reduces
> the effort the Flink Master has to do whenever global coordination is
> required, i.e., during checkpointing. And the network stack in the
> task managers is optimized to take advantage of locality, whenever
> possible.
>
> On the other hand, if you have a lot of pressure on the heap (e.g.,
> because you are using a heap-based state backend), then having more,
> smaller task managers can reduce latency by reducing the impact of
> garbage collection pauses.
>
> I'm sure I've overlooked some factors, but the bottom line appears to
> be that there's no one-size-fits-all answer.
>
> David
>
> On Wed, Sep 25, 2019 at 5:43 PM Navneeth Krishnan
> <reachnavneeth2@gmail.com> wrote:
> >
> > Thanks Terry, the reason why I asked this is because somewhere I saw
> running one slot per container is beneficial. I couldn’t find the where I
> saw that.
> > Also I think running it with multiple slots will reduce IPC since some
> of the data will be processed writhing the same JVM.
> >
> > Thanks
> >
> > On Wed, Sep 25, 2019 at 1:16 AM Terry Wang <zjuwangg@gmail.com> wrote:
> >>
> >> Hi, Navneeth,
> >>
> >> I think both is ok.
> >> IMO, run one container with number of slots same as virtual cores may
> be better for slots can share the Flink Framework and thus reduce memory
> cost.
> >>
> >> Best,
> >> Terry Wang
> >>
> >>
> >>
> >> > 在 2019年9月25日,下午3:26,Navneeth Krishnan <reachnavneeth2@gmail.com>
写道:
> >> >
> >> > Hi All,
> >> >
> >> > I’m currently running flink on amazon ecs and I have assigned task
> slots based on vcpus per instance. Is it beneficial to run a separate
> container with one slot each or one container with number of slots same as
> virtual cores?
> >> >
> >> > Thanks
> >>
>


-- 
Thanks & Regards
Sri Tummala

Mime
View raw message