flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chakravarthy varaga <chakravarth...@gmail.com>
Subject Re: deploying flink in AWS - some teething issues
Date Thu, 30 Mar 2017 12:39:54 GMT

    With regards to logging (both Flink & application specific logs) within
the container, are there best practices that you know of to get the logs to
a centralized locations..
    For e.g. the flink TM's log are local inside the container and I don't
wish to write to shared/mounted volumes, this means that I have to run
separate daemon running inside the container along with the TM to transport
these logs to another server or so...
    + I don't see that flink provides support for syslog to be able to
connect to rsyslog etc.,

    Can you please advice a way to go here?

Best Regards

On Wed, Mar 29, 2017 at 11:33 AM, Patrick Lucas <patrick@data-artisans.com>

> For 1., I think the standard approach would be to specify from without
> what the heap size should be. If you want an *x* MB heap, you could set
> your container memory limit to 1.3 * *x* or so (to account for overhead)
> and set taskmanager.heap.mb: *x* in your config.
> The other way around—e.g. from inside the container determine its memory
> limit and divide it by 1.3—sounds interesting though, so please share if
> you have success with that.
> For 2. I don't think there's really a good way yet to monitor the health
> of containerized jobs directly, so probably your best bet is to watch the
> job's metrics from outside the Flink cluster.
> --
> Patrick Lucas
> On Wed, Mar 29, 2017 at 10:58 AM, Chakravarthy varaga <
> chakravarthyvp@gmail.com> wrote:
>> Hi,
>>     Any updates here? I'm sure many would have faced similar issues like
>> these, any help here is highly appreciated.
>> Best Regards
>> CVP
>> On Tue, Mar 28, 2017 at 5:47 PM, Chakravarthy varaga <
>> chakravarthyvp@gmail.com> wrote:
>>> Hi Team,
>>>    If the flink cluster is conainerized and managed through by a
>>> container orchestrator,
>>>     1.  the orchestrator allocates resources for each JM. TM etc., say
>>> if the container (TM) needs to run with 2G RAM, how should this allocation
>>> be honoured by the TM when its JVM starts. I'm thinking of wrapping up a
>>> script that determines the resource allocation for the container and writes
>>> the flink-conf.yaml before the TM starts the process. Is this the way to go?
>>>     2. The container orchestrator looks at health of the containers and
>>> is however unaware of the job health status runnning inside the
>>> container/cluster. How should this be determined?
>>> Best Regards
>>> CVP

View raw message