flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Lucas <patr...@data-artisans.com>
Subject Re: deploying flink in AWS - some teething issues
Date Wed, 29 Mar 2017 10:33:33 GMT
For 1., I think the standard approach would be to specify from without what
the heap size should be. If you want an *x* MB heap, you could set your
container memory limit to 1.3 * *x* or so (to account for overhead)
and set taskmanager.heap.mb:
*x* in your config.

The other way around—e.g. from inside the container determine its memory
limit and divide it by 1.3—sounds interesting though, so please share if
you have success with that.

For 2. I don't think there's really a good way yet to monitor the health of
containerized jobs directly, so probably your best bet is to watch the
job's metrics from outside the Flink cluster.

--
Patrick Lucas

On Wed, Mar 29, 2017 at 10:58 AM, Chakravarthy varaga <
chakravarthyvp@gmail.com> wrote:

> Hi,
>
>     Any updates here? I'm sure many would have faced similar issues like
> these, any help here is highly appreciated.
>
> Best Regards
> CVP
>
> On Tue, Mar 28, 2017 at 5:47 PM, Chakravarthy varaga <
> chakravarthyvp@gmail.com> wrote:
>
>> Hi Team,
>>
>>    If the flink cluster is conainerized and managed through by a
>> container orchestrator,
>>
>>     1.  the orchestrator allocates resources for each JM. TM etc., say if
>> the container (TM) needs to run with 2G RAM, how should this allocation be
>> honoured by the TM when its JVM starts. I'm thinking of wrapping up a
>> script that determines the resource allocation for the container and writes
>> the flink-conf.yaml before the TM starts the process. Is this the way to go?
>>
>>     2. The container orchestrator looks at health of the containers and
>> is however unaware of the job health status runnning inside the
>> container/cluster. How should this be determined?
>>
>> Best Regards
>> CVP
>>
>>
>>
>

Mime
View raw message