flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oytun Tez <oy...@motaword.com>
Subject Re: Running flink on AWS ECS
Date Sat, 28 Sep 2019 13:13:29 GMT
I said to myself I'll take another look at Kinesis Analytics, but this is
the state of it:
[image: image.png]


Flink version is 1.6, which is way behind and lacking many good features
and reliability improvements.


---
Oytun Tez

*M O T A W O R D*
The World's Fastest Human Translation Platform.
oytun@motaword.com — www.motaword.com


On Sat, Sep 28, 2019 at 9:04 AM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:

> Aws already has auto scale flink cluster it’s called Kinesis Data
> Analytics just add your flink Jar to Kinesis Sql analytics that’s all , aws
> will auto provision a flink cluster and do the admin part for you.
>
> On Saturday, September 28, 2019, David Anderson <david@ververica.com>
> wrote:
>
>> I believe there can be advantages and disadvantages in both
>> directions. For example, fewer containers with multiple slots reduces
>> the effort the Flink Master has to do whenever global coordination is
>> required, i.e., during checkpointing. And the network stack in the
>> task managers is optimized to take advantage of locality, whenever
>> possible.
>>
>> On the other hand, if you have a lot of pressure on the heap (e.g.,
>> because you are using a heap-based state backend), then having more,
>> smaller task managers can reduce latency by reducing the impact of
>> garbage collection pauses.
>>
>> I'm sure I've overlooked some factors, but the bottom line appears to
>> be that there's no one-size-fits-all answer.
>>
>> David
>>
>> On Wed, Sep 25, 2019 at 5:43 PM Navneeth Krishnan
>> <reachnavneeth2@gmail.com> wrote:
>> >
>> > Thanks Terry, the reason why I asked this is because somewhere I saw
>> running one slot per container is beneficial. I couldn’t find the where I
>> saw that.
>> > Also I think running it with multiple slots will reduce IPC since some
>> of the data will be processed writhing the same JVM.
>> >
>> > Thanks
>> >
>> > On Wed, Sep 25, 2019 at 1:16 AM Terry Wang <zjuwangg@gmail.com> wrote:
>> >>
>> >> Hi, Navneeth,
>> >>
>> >> I think both is ok.
>> >> IMO, run one container with number of slots same as virtual cores may
>> be better for slots can share the Flink Framework and thus reduce memory
>> cost.
>> >>
>> >> Best,
>> >> Terry Wang
>> >>
>> >>
>> >>
>> >> > 在 2019年9月25日,下午3:26,Navneeth Krishnan <reachnavneeth2@gmail.com>
写道:
>> >> >
>> >> > Hi All,
>> >> >
>> >> > I’m currently running flink on amazon ecs and I have assigned task
>> slots based on vcpus per instance. Is it beneficial to run a separate
>> container with one slot each or one container with number of slots same as
>> virtual cores?
>> >> >
>> >> > Thanks
>> >>
>>
>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
>

Mime
View raw message