flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ritesh Kumar Singh <riteshoneinamill...@gmail.com>
Subject Re: Task Manager metrics per job on Flink 0.9.1
Date Tue, 26 Jan 2016 20:22:24 GMT
I didn't know these stats were collected. Thanks for telling :)
In that case, it should definitely be a feature which can be enabled via
config files.

*Ritesh Kumar Singh,*
*https://riteshtoday.wordpress.com/* <https://riteshtoday.wordpress.com/>

On Tue, Jan 26, 2016 at 8:22 PM, Pieter Hameete <phameete@gmail.com> wrote:

> Hi Ritesh,
> thanks for the response! The metrics are already being gathered though, so
> I think it would be nice to have a configuration/option to log them
> somewhere. It doesnt have to be enabled by default, and I dont think it
> should degrade the performance very much. It looks like the metrics are
> currently sent with each heartbeat by default already. Your Web UI probably
> hangs because it has to update all the graphs on every heartbeat, when you
> have many task managers that will be heavy on your computer :-)
> - Pieter
> 2016-01-26 20:17 GMT+01:00 Ritesh Kumar Singh <
> riteshoneinamillion@gmail.com>:
>> Going by the list in the latest documentation
>> <https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html>
>> for Flink 0.10.1 release, memory and cpu stats are not stored. Neither is
>> the time spent on garbage collection stored anywhere.
>> In my opinion, trying to store these metrics will degrade the performance
>> of jobs too. And so its basically a trade off between performance and
>> computation cost. For me, the web ui hangs even for the current set of
>> parameters :(
>> *Ritesh Kumar Singh,*
>> *https://riteshtoday.wordpress.com/* <https://riteshtoday.wordpress.com/>
>> On Tue, Jan 26, 2016 at 7:16 PM, Pieter Hameete <phameete@gmail.com>
>> wrote:
>>> Hi people!
>>> A lot of metrics are gathered for each TaskManager every few seconds.
>>> The web UI shows nice graphs for some of these metrics too.
>>> I would like to make graphs of the memory and cpu usage, and the time
>>> spent on garbage collection for each job. Because of this I am wondering if
>>> the metrics are also stored somewhere, or if there is an option to enable
>>> storing the metrics per job.
>>> In the configuration documentation I could not find such an option. Is
>>> this possible in version 0.9.1 of Flink? If not: is it possible in Flink
>>> 0.10.1 or is it possible to request or develop such a feature?
>>> Thank you for your help and kind regards,
>>> Pieter

View raw message